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The  federal  government  will  invest  $80  billion  in  fiscal  year  1999  on 
research  and  development  (r&d)  performed  by  government  scientists  and 
through  grants,  contracts,  or  other  agreements  with  universities, 
corporations,  small  businesses,  and  other  members  of  the  research 
community.  The  results  of  this  research  can  become  the  impetus  for 
change  in  federal  policies,  regulations,  and  programs.  For  example, 
research  in  medicine,  aviation,  and  agriculture  affect  policy  or  regulatory 
decisions  for  drug  and  airline  safety  and  pesticide  use.  Thus,  the  Congress 
and  the  public  rely  on  federal  agencies  to  fund  and  conduct  research  that 
produces  high-quality  results.  To  help  ensure  the  quality  and  integrity  of 
the  research,  U.S.  science  has  traditionally  relied  on  independent  reviews 
by  peers. 

This  report  responds  to  your  request  that  we  study  the  peer  review  and 
other  quality  assurance  processes  that  federal  agencies  use  in  conducting 
scientific  research  and  development.  Specifically,  as  agreed  with  your 
offices,  we  (1)  define  what  is  meant  by  peer  review,  (2)  describe  the 
federal  government’s  peer  review  policy,  (3)  describe  the  peer  review 
practices  of  12  federal  agencies  that  conduct  scientific  research, 

(4)  describe  other  agency  quality  assurance  reviews; ,  and  (5)  identify 
which  research  is  not  subjected  to  review. 

On  the  basis  of  discussions  with  your  staffs,  we  included  the  following  12 
federal  agencies  in  our  review:  3  entities  within  the  Department  of 
Agriculture,  the  Agricultural  Research  Service,  the  Cooperative  State 
Research,  Education,  and  Extension  Service,  and  the  Forest  Service;  2 
entities  within  the  Department  of  Commerce,  the  National  Institute  of 
Standards  and  Technology  and  the  National  Oceanic  and  Atmospheric 
Administration;  the  Department  of  Energy;  the  Environmental  Protection 
Agency;  the  Department  of  Health  and  Human  Services’  National  Institutes 
of  Health;  the  Department  of  the  Interior’s  U.S.  Geological  Survey;  the 
National  Aeronautics  and  Space  Administration;  the  National  Science 
Foundation;  and  the  Federal  Aviation  Administration  within  the 
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Results  in  Brief 


Department  of  Transportation.  In  this  report,  we  refer  to  the  Department 
of  Energy,  the  bureaus,  and  independent  agencies  as  agencies.  These 
agencies  account  for  more  than  90  percent  of  the  federal  r&d  budget  in 
fiscal  year  1999,  excluding  the  Department  of  Defense. 


There  is  no  written  govemmentwide  definition  of  peer  review.  Officials  at 
the  Office  of  Science  and  Technology  Policy  and  at  the  agencies  we 
contacted  generally  concur  that  peer  review  is  defined  as  a  process  that 
includes  an  independent  assessment  of  the  technical,  scientific  merit  of 
research  by  peers  who  are  scientists  with  knowledge  and  expertise  equal 
to  that  of  the  researchers  whose  work  they  review. 

There  is  no  uniform  federal  policy  for  conducting  peer  reviews.  Through 
annual  budget  guidance  to  federal  agencies,  the  Office  of  Science  and 
Technology  Policy  and  the  Office  of  Management  and  Budget  encourage 
funding  of  research  projects  that  are  peer  reviewed  over  those  that  are  not 
reviewed.  Officials  at  the  Office  of  Science  and  Technology  Policy  said 
that  peer  review  practices  should  not  be  dictated  uniformly  for  every 
agency  or  for  all  types  of  federally  funded  research.  Rather,  the  practices 
should  be  tailored  to  agency  missions  and  type  of  research.  Each  of  the  12 
agencies  that  we  contacted  had  a  variety  of  policies,  orders,  or  other 
internal  guidance  regarding  the  conduct  of  peer  review. 

To  varying  degrees,  the  12  agencies  use  peer  review  to  (1)  assess  the  merit 
of  competitive  and  noncompetitive  research  proposals,  (2)  determine 
whether  to  continue  or  renew  research  projects,  (3)  evaluate  the  results  of 
the  research  prior  to  the  publication  of  those  results,  (4)  establish  annual 
budget  priorities  for  research  programs,  and  (5)  evaluate  program  and 
scientist  performance.  All  of  the  agencies  use  peer  review  to  assess 
competitive  research  proposals.  The  methods  for  conducting  peer  reviews 
vary  among  and  within  the  agencies.  For  example,  the  agencies  select  peer 
reviewers  from  academia,  private  industry,  and  government  and  obtain 
review  comments  by  mail  and  through  panel  meetings,  site  visits,  and 
workshops  or  a  combination  of  methods. 

Most  of  the  agencies  that  we  reviewed  also  use  reviews  by  agency 
supervisors  or  program  managers  to  assess  the  quality  of  research 
proposals,  to  check  the  quality  of  in-progress  research,  and  to  evaluate 
program  performance.  Generally,  these  quality  assurance  reviews  are  not 
considered  independent  assessments — ,  a  key  criterion  in  the  peer  review 
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process.  These  quality  assurance  reviews  occur  at  both  the  project  and 
program  levels. 

While  agencies  reported  that  almost  all  research  is  reviewed  either 
through  peer  reviews  or  other  quality  assurance  reviews,  a  small  amount 
of  research  may  not  be  reviewed  by  the  agencies  in  certain  circumstances. 
Examples  of  research  that  may  be  funded  without  being  reviewed  include 
projects  that  are  congressionally  mandated  or  projects  that  use  widely 
accepted  methodologies. 


Background 


Scientific  research  has  traditionally  been  considered  to  be  valuable  to 
society.  The  knowledge  gained  from  federally  supported  research  leads  to 
the  development  of  new  products  and  processes.  Research  results  also 
provide  information  needed  to  make  policy  decisions.  The  Congress  is 
concerned  that  quality  research  is  used  to  provide  a  rational  basis  for 
federal  rule-  making,  as  with  regulations  established  to  protect  human 
health  and  safety  and  the  environment.  Peer  review  has  been  used  to  judge 
the  quality  of  science  for  over  3  centuries.  Historically,  peer  review  has 
been  used  extensively  in  the  selection  of  proposed  research  projects  and 
to  a  lesser  extent  to  evaluate  r&d  programs  and  their  likely  effects. 

The  Congress  and  the  President  have  taken  steps  to  help  improve  the 
management  of  and  accountability  for  federal  r&d  spending.  In  1976,  the 
Congress  established  the  Office  of  Science  and  Technology  Policy,  which 
serves  as  a  source  of  scientific,  engineering,  and  technological  analysis 
and  judgment  for  the  President  and  assists  him  in  providing  leadership  and 
coordination  for  federal  r&d  programs.1  In  November  1993,  by  executive 
order,  the  President  also  established  the  Cabinet-level  National  Science 
and  Technology  Council  to  coordinate  federal  r&d  and  to  establish  clear 
national  goals  for  science  and  technology  investments.2  Through  this 
Council,  the  Office  of  Science  and  Technology  Policy  helps  the  President 
coordinate  science,  space,  and  technology  policy  and  programs  across  the 
federal  government  and  leads  an  interagency  effort  to  develop  and 
implement  science  and  technology  policies  and  budgets  across  federal 
agencies. 

In  1993,  the  Congress  passed  the  Government  Performance  and  Results 
Act  to  encourage  efficiency,  effectiveness,  and  accountability  in  federal 


National  Science  and  Technology  Policy,  Organization,  and  Priorities  Act  of  1976,  P.L.  94-282  (May  11, 
1976). 

2E.0. 12881,  (Nov.  23, 1993). 


Page  3 


GAO/RCED-99-99  Federal  Research 


programs.  The  act  requires  federal  agencies  to  produce  strategic 
performance  plans  with  annual  targets  and  annual  performance  reports 
that  explain  whether  those  targets  have  been  met.  A  recent  report  by  the 
Committee  on  Science,  Engineering,  and  Public  Policy  concluded  that  the 
most  effective  means  of  evaluating  federally  funded  research  programs  is 
expert  review.3  The  best  known  form  of  expert  review  is  peer  review;  two 
other  forms  are  relevance  review  that  examines  whether  the  research 
program  focuses  on  an  agency’s  mission  and  international  benchmarking 
that  determines  whether  the  research  is  at  the  forefront  of  scientific  and 
technological  knowledge. 


Definition  of  Peer 
Review 


There  is  no  written  definition  of  peer  review  that  applies  across  the  federal 
government.  Officials  at  the  Office  of  Science  and  Technology  Policy 
described  peer  review  as  a  merit-based  process  or  independent  merit 
assessment,  generally  used  in  decisions  about  which  research  projects  to 
fund.  Individual  agencies  define  peer  review  somewhat  differently; 
however,  all  of  the  agencies’  definitions  or  descriptions  of  peer  review 
contained  the  fundamental  concept  of  a  review  of  technical  or  scientific 
merit  by  individuals  with  sufficient  technical  competence  and  no 
unresolved  conflict  of  interest.4 

Peers  generally  are  considered  to  be  scientists  or  engineers  who  have 
qualifications  and  expertise  equivalent  to  those  of  the  researcher  whose 
work  they  review.  In  addition,  peers  must  be  capable  of  making  an 
independent  judgment  of  the  merits  and  relevance  of  the  research. 

Officials  at  the  Office  of  Science  and  Technology  Policy  said  that  peers  do 
not  have  to  be  external  to  the  funding  agency,  as  long  as  there  is  no 
unresolved  conflict  of  interest. 


Peer  Review  Policy 


Officials  at  the  Office  of  Science  and  Technology  Policy  said  that  there  is 
no  govemmentwide  legislation  or  policy  that  requires  agencies  to  conduct 
peer  reviews  or  dictates  how  the  reviews  should  be  conducted.  However, 
the  office  advocates  the  use  of  peer  review  and  provides  guidance  to 
agencies  on  the  use  of  peer  review  to  assess  the  quality  of  research.  For 
example,  the  office  works  with  the  Office  of  Management  and  Budget  to 
promote  peer  review.  Beginning  with  the  fiscal  year  1996  budget  cycle,  the 


Evaluating  Federal  Research  Programs:  Research  and  the  Government  Performance  and  Results  Act. 
(Feb.  1999),  Washington,  D.C.  The  Committee  on  Science,  Engineering,  and  Public  Policy  is  a  joint 
committee  of  the  National  Academy  of  Sciences,  the  National  Academy  of  Engineering,  and  the 
Institute  of  Medicine. 

4Individual  agency  definitions  of  peer  review  are  included  in  appendixes  I  through  XII. 
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two  offices  have  jointly  provided  annual  direction  to  agencies  encouraging 
them  to  emphasize  the  funding  of  peer-reviewed  research  over 
nonpeer-reviewed  research.  As  a  result,  Office  of  Science  and  Technology 
Policy  officials  said  that  agencies  have  shifted  funding  toward  more 
peer-reviewed  research.  The  guidance  also  encourages  agencies  to  solicit 
proposals  from  many  researchers  on  how  to  solve  research  problems. 

Officials  at  the  Office  of  Science  and  Technology  Policy  said  that  agencies’ 
peer  review  practices  should  be  flexible  and  tailored  to  agency  missions 
and  type  of  research,  and  that  specific  uniform  practices  should  not  be 
dictated  for  every  agency  or  all  federally  funded  research.  A  variety  of  peer 
review  methods  is  viewed  by  the  agencies  as  both  appropriate  and 
essential,  reflecting  the  varying  nature  of  the  research  and  its  purposes, 
the  differences  in  research  timelines,  the  broad  spectrum  of  r&d 
performers,  and  the  varying  funding  mechanisms,  such  as  grants, 
contracts,  and  cooperative  agreements.  For  example,  agency  officials 
noted  that  specific  peer  reviews  can  have  quite  different  purposes  such  as 
the  review  of  research  results  prior  to  publication  versus  the  review  of 
agency  or  program  performance. 

While  peer  review  has  come  to  be  viewed  by  some  observers  as  the  best 
assurance  that  quality  criteria  will  prevail  over  social,  economic,  and 
political  considerations,  others  view  peer  review  as  an  element  of  elitism 
in  science  that  tends  to  discount  such  concerns  as  economic 
considerations.  Officials  at  the  National  Science  Foundation  said  that 
under  certain  conditions,  over  emphasis  on  peer  review  for  funding 
decisions  may  discourage  funding  of  innovative  research  because  peers 
generally  tend  to  view  research  somewhat  conservatively.  Officials  at  the 
Office  of  Science  and  Technology  said  that  this  phenomenon  has  been 
noted  by  other  agencies,  and  the  Office  is  initiating  an  interagency  effort  to 
examine  how  peer  review  practices  (and  other  federal  r&d  policies)  might 
better  facilitate  innovation. 

A  July  1996  National  Science  and  Technology  Council  report  emphasized 
the  need  for  flexibility  in  implementing  peer  review  and  described 
agencies’  merit  review  with  peer  evaluation  as  one  strategy  for  evaluating 
the  performance  of  fundamental  science  programs.6  Underlying  principles 
stated  in  the  report  included  the  following: 


5 A  report  from  the  Subcommittee  on  Research,  Committee  on  Fundamental  Science.  National  Science 
and  Technology  Council.  “Assessing  Fundamental  Science.”  Washington,  D.C.  (July  1996). 
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“Science  agencies  must  devise  assessment  strategies  that  are  appropriate  to  the  nature  of 
scientific  processes  and  to  the  enabling  role  of  fundamental  science  in  support  of 
over-arching  national  goals...  [the  strategies]  should  be  designed  to...respond  to  surprises, 
pursue  detours,  and  revise  program  agendas  in  response  to  new  scientific  information  and 
technical  opportunities  essential  to  the  future  well-being  of  all  our  people.” 

All  the  agencies  that  we  contacted  identified  policies,  orders,  or  other 
internal  guidance  regarding  the  conduct  of  peer  review.  Some  of  these 
policies  are  legislatively  mandated.  For  example,  the  Agricultural 
Research,  Extension,  and  Education  Reform  Act  of  1998  requires  the  U.S. 
Department  of  Agriculture  to  peer  review  Department-funded  research.6 
The  law  also  requires  grantees  to  arrange  for  a  peer  review  of  special 
grants  mandated  by  the  Congress.  Overall,  we  found  that  eight 
agencies — the  Agricultural  Research  Service,  the  Cooperative  State 
Research,  Education,  and  Extension  Service,  the  National  Institute  of 
Standards  and  Technology,  the  National  Institutes  of  Health,  the 
Department  of  Energy,  the  Environmental  Protection  Agency,  the  National 
Aeronautics  and  Space  Administration,  and  the  Federal  Aviation 
Administration — have  laws  or  regulations  that  require  peer  review  of 
competitively  selected  grant  proposals. 


pppr  •  Practices  The  peer  review  Practices  differ  among  and  within  federal  research 

JTCCI  ivt;  VIC  w  ri  delicti  agencjes  in  two  ways.  First,  agencies  use  peer  review  in  varying  degrees  to 

Vary  assess  the  merit  of  research  at  different  stages  in  the  research  process, 

including  selecting  research  projects  for  funding;  monitoring  in-progress 
research;  and  evaluating  research  products  prior  to  publication.  Second, 
the  implementation  of  the  peer  review  process  varies.  The  following 
highlights  the  extent  to  which  agencies  use  peer  review  and  some  of  the 
various  ways  they  implement  peer  review.  Appendixes  I  through  XII 
provide  descriptions  of  the  agencies’  peer  review  practices,  to  the  extent 
that  peer  review  is  used,  for  each  of  the  12  agencies  included  in  our 
review. 


Reviews  of  Research 
Proposals 


All  the  agencies  conduct  peer  reviews  to  help  determine  which 
competitive  research  proposals  to  fund.  All  12  agencies  also  use  peer 
review  to  help  determine  funding  for  at  least  a  portion  of  their  other 
research,  including  peer  reviews  of  the  agencies’  intramural  or  internal 
research  proposals  or  plans.  The  agencies  use  a  combination  of  external 
and  internal  reviewers  with  subject  matter  expertise.  However,  Federal 


r,P.L  105-185,  (June  23, 1998). 
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Reviews  of  In-Progress 
Research 


Aviation  Administration  officials  said  that  their  peer  reviewers  are 
primarily  agency  employees  who  are  not  involved  in  the  project  but  have 
the  required  subject  matter  expertise. 

The  agencies  conduct  the  peer  reviews  by  mail,  panels  or  committees,  or  a 
combination  of  methods.  They  also  differ  in  the  number  of  reviewers  used 
in  the  process.  For  example,  the  Cooperative  State  Research,  Education, 
and  Extension  Service  uses  panels  of  outside  experts  to  review 
competitive  research  proposals.  One  Cooperative  State  Research, 
Education,  and  Extension  Service  program,  the  National  Research 
Initiative,  uses  panels  that  meet  throughout  the  year.  Each  panel  member 
reviews  about  30  research  proposals  and  provides  written  comments  on 
about  20  of  the  proposals.  Mail  reviews  and  ad  hoc  reviewers  are  also  used 
when  additional  expertise  is  needed.  In  contrast,  the  National  Aeronautics 
and  Space  Administration  generally  conducts  reviews  by  mail  to  obtain 
specialized  expertise  on  technical  issues  and  approaches,  followed  by 
reviews  by  panels  of  7  to  10  experts  with  broader  perspectives  to  reconcile 
differences  among  the  mail  reviewers’  comments. 

The  agencies  use  various  criteria  to  assess  proposed  research,  including 
technical  or  scientific  merit,  relevance  to  agency  mission  and  priorities, 
and  the  qualifications  of  the  researcher.  Agency  officials  responsible  for 
selecting  research  for  funding  generally  consider  the  peer  reviewers’ 
advice  or  recommendations  along  with  other  financial  and  management 
factors  to  make  their  funding  decisions. 

In  some  circumstances,  agencies  use  these  same  types  of  peer  review 
processes  to  assess  the  merit  of  research  that  is  not  funded  through 
competitive  selection,  generally  this  research  is  internal  to  the  agencies. 
For  example,  the  Forest  Service  conducts  peer  reviews  of  its  scientists’ 
research  study  plans  prior  to  approval  by  project  leaders.  The  U.S. 
Geological  Survey  conducts  peer  reviews  of  all  intramural  project 
proposals  to  ensure  technical  quality  prior  to  final  approval  and 
implementation.  At  the  Agricultural  Research  Service,  internal  peers,  who 
are  not  involved  in  the  funding  decision,  review  detailed  project  plans. 


|  While  all  12  agencies  provided  examples  of  peer  reviews  of  research  that 
are  in-progress,  there  is  much  variation  in  the  frequency  and  purpose  of 
those  reviews.  The  purposes  of  these  reviews  include  assessments  of 
research  projects  to  determine  if  funding  should  be  renewed  or  to  assess 
the  progress  of  on-going  research  at  the  program  level,  or  at  research 
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Reviews  of  Publications 


Other  Peer  Reviews 


stations  or  laboratories.  For  example,  in-progress  Cooperative  State 
Research,  Education,  and  Extension  Survey  projects  that  are  assessed  for 
renewal  within  2  to  3  years  compete  with  new  proposals  in  the  same  merit 
review  process.  If  the  review  panel  considers  the  research  progress  to  be 
unsatisfactory,  the  research  project  will  not  receive  additional  funding. 

The  National  Aeronautics  and  Space  Administration  requires  that  for  any 
project  that  continues  for  more  than  3  years,  the  researcher  must  submit  a 
new  proposal,  which  is  subject  to  external  peer  review. 

The  National  Institutes  of  Health  uses  boards  of  outside  experts  to  peer 
review  its  on-going  intramural  research,  and  the  Agricultural  Research 
Service  convenes  panels  in  a  workshop  format  to  review  intramural 
research  projects.  The  U.S.  Geological  Survey’s  peer  review  guidance 
requires  that  on-going  programs  undergo  external  peer  review  about  every 
5  years.  The  National  Oceanic  and  Atmospheric  Administration  also  has 
peer  reviews  of  the  work  funded  and  completed  at  its  laboratories.  Every  3 
to  5  years,  qualified  peers  evaluate  program  accomplishments  and  impacts 
in  the  context  of  the  resources  invested  in  them.  At  the  National  Institute 
of  Standards  and  Technology,  the  National  Research  Council’s  Board  on 
Assessment  of  the  Institute’s  programs  annually  conducts  reviews  of  the 
technical  quality  and  relevance  of  planned,  ongoing,  and  completed 
laboratory  programs.  On  a  cyclical  basis,  the  Forest  Service  Deputy  Chiefs 
Program  Reviews  use  external  peers  in  the  evaluations  of  a  research 
station’s  overall  program  to  improve  program  results. 


Generally,  the  agency  officials  said  that  their  agencies  encourage  their 
scientists  to  publish  research  results  in  professional  journals  that  conduct 
peer  reviews  of  manuscripts  prior  to  accepting  them  for  publication.  Some 
agencies  also  peer  review  draft  work  products,  manuscripts,  or  other 
research  results  prior  to  publication.  For  example,  the  Agricultural 
Research  Service,  the  Forest  Service,  the  Environmental  Protection 
Agency,  the  National  Oceanic  and  Atmospheric  Administration’s  National 
Marine  Fisheries  Service,  the  National  Institutes  of  Health,  and  the  U.S. 
Geological  Survey  review  their  scientists’  manuscripts  prior  to  publication 
in  in-house  technical  reports  or  professional  journals. 


Agencies  also  use  peer-review  techniques  to  aid  priority  setting,  program 
development,  and  personnel  evaluation.  For  example,  the  Federal  Aviation 
Administration’s  annual  programming  and  budgeting  process  includes 
program-level  peer  reviews  of  proposed  and  in-progress  research  by  a 
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30-member,  legislatively  authorized  Research,  Engineering,  and 
Development  Advisory  Committee,  established  under  the  Aviation  Safety 
Research  Act  of  1988,  as  amended.  This  process  results  in 
recommendations  about  the  merit  of  the  research  and  funding  priorities  in 
the  Federal  Aviation  Administration’s  six  program  areas.  The 
Environmental  Protection  Agency’s  Science  Advisory  Board,  consisting  of 
scientists,  engineers,  and  economists  from  academia,  industry,  and 
environmental  communities,  reviews  the  technical  basis  for  the  agency’s 
science  policy  positions,  including  scientific  documents  used  to  support 
environmental  regulations.  Independent  scientists  provide  advice  to  the 
Department  of  Energy  on  the  quality,  relevance,  and  productivity  of  its 
laboratory  research,  in  conjunction  with  program  reviews  and  advisory 
committee  oversight.  Last,  the  Agricultural  Research  Service  and  the 
Forest  Service  also  consider  the  required  peer  reviews  of  their  individual 
scientist’s  research  accomplishments  to  constitute  additional  checks  of 
the  quality  of  their  research. 


Agencies  Use  of  Other 
Quality  Assurance 
Reviews 


In  addition  to  peer  reviews,  most  agencies  also  conduct  various  types  of 
internal  reviews  as  checks  on  the  quality  of  their  research.  These  reviews 
are  generally  conducted  by  supervisors  or  managers  and  are,  therefore, 
not  independent  reviews  of  the  research.  Agencies  conduct  these  quality 
assurance  reviews  to  assess  the  merit  of  proposed  research,  to  assess  the 
progress  of  on-going  research,  and  to  evaluate  research  results.  These 
reviews  occur  at  both  the  project  and  program  level.  TThe  following  are 
examples  of  agencies’  internal  reviews. 


Scientists  at  the  Cooperative  State  Research,  Education,  and  Extension 
Service  review  noncompetitive  proposals  for  formula-funded  awards  and 
for  congressionally  mandated  research.  Cooperative  State  Research, 
Education,  and  Extension  Service  scientists  also  review  annual  progress 
reports  prepared  by  the  researchers.  Likewise,  Agricultural  Research 
Service  line  managers  review  annual  project  progress  reports  submitted  by 
the  Service’s  lead  scientists.  These  reviews  may  lead  to  revisions  of 
program  plans.  National  Institute  of  Standards  and  Technology  officials 
track  the  results  of  research  funded  by  the  Advanced  Technology 
Program,  using  quarterly  progress  reports  and  annual  meetings  with 
recipients.  The  National  Aeronautics  and  Space  Administration  project 
managers  and  staff  frequently  comment  on  technical,  management,  and 
financial  aspects  of  a  proposal,  since  science  reviewers  may  not  be 
qualified  experts  in  these  fields.  The  Federal  Aviation  Administration’s 
Civil  Aeromedical  Institute,  which  conducts  mostly  intramural  research, 
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reviews  the  technical  details  of  proposed  projects.  According  to  Federal 
Aviation  Administration  officials,  the  Institute  employs  some  of  the 
world’s  best  scientists,  so  the  number  of  outside  experts  is  limited. 
Research  performed  under  Federal  Aviation  Administration  procurement 
contracts  is  reviewed  during  the  annual  programming  and  budgeting 
process.  Subsequently,  when  a  decision  is  made  to  fund  research  through 
a  contract,  Federal  Aviation  Administration  technical  staff  monitor  the 
work. 

In  addition,  the  Agricultural  Research  Service,  the  Cooperative  State 
Research,  Education,  and  Extension  Service,  the  Forest  Service,  the 
National  Institute  of  Standards  and  Technology,  the  National  Institutes  of 
Health,  the  National  Oceanic  and  Atmospheric  Administration,  the 
Department  of  Energy,  and  the  National  Science  Foundation  use  advisory 
committees  to  help  establish  research  programs  or  priorities.  For  example, 
section  103  of  the  Agricultural  Research,  Extension,  and  Education 
Reform  Act  of  1998  requires  that  the  National  Agricultural  Research, 
Extension,  Education,  and  Economics  Advisory  Board  annually  review  the 
Department  of  Agriculture’s  research  priority  setting.  A  Forest  Research 
Advisory  Committee  to  the  Secretary  of  Agriculture,  made  up  of  university, 
industry,  and  interest  group  representatives,  also  reviews  and  comments 
on  the  Forest  Service’s  current  and  future  research  priorities.  The  National 
Institute  of  Standards  and  Technology  statutory  Visiting  Committee  on 
Advanced  Technology  meets  quarterly  to  review  agency  research  policies, 
budget,  organization,  and  programs. 

A  few  agency  officials  also  discussed  their  use  of  peer  review  in  their 
efforts  to  comply  with  the  Results  Act  requirements  for  reporting 
performance  measures,  goals,  and  outcomes  of  their  research.  For 
example,  the  Federal  Aviation  Administration’s  annual  programming  and 
budget  process  includes  reviews  of  research  outcomes  and  outputs  and  is 
performed  in  the  context  of  the  Results  Act.  A  Department  of  Agriculture 
report  stated  that  the  Cooperative  State  Research,  Education,  and 
Extension  Service’s  Results  Act  report  might  potentially  provide  a 
mechanism  to  more  systematically  monitor  and  report  research 
performance  and  accomplishments.7  The  National  Science  Foundation  is 
trying  to  track  the  results  of  its  research  efforts  through  annual  and  final 
reports  on  each  project  and  plans  to  follow-up  for  years  after  completion 
to  identify  retrospective  impacts  from  the  research. 


7Quality  of  Agricultural  Research.  Report  of  the  Research,  Education,  and  Economics  Quality  Research 
Initiative  Task  Force.  (Jan.  1996). 
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Research  Not  Subject 
to  Review 


While  the  agencies  said  that  they  conduct  either  peer  reviews  or  other 
quality  reviews  for  almost  all  of  their  research,  there  are  small  amounts  of 
research  that  may  not  be  reviewed.  For  example,  officials  from  the 
Agricultural  Research  Service,  the  National  Institute  of  Standards  and 
Technology,  and  the  Department  of  Energy  said  that  they  did  not  always 
review  specific  research  proposals  when  the  agency  is  directed  to  perform 
the  research.  In  addition,  research  performed  jointly  with  outside  entities, 
such  as  with  cooperative  research  and  development  agreements,  may  not 
always  be  fully  subjected  to  review,  depending  on  factors  such  as  the 
nature  of  the  partnership  and  the  presence  of  proprietary  information.  The 
Congress  directs  the  Agricultural  Research  Service  to  allocate  about 
$10  million  annually  to  external  researchers  for  specific  cooperative 
agreements.  While  the  Agricultural  Research  Service  does  not  usually  peer 
review  this  research  prior  to  funding,  it  does  review  progress  reports  that 
the  researcher  is  required  to  submit  annually.  Officials  from  the  National 
Institutes  of  Health  said  that  supplemental  funding  requested  to  carry  on 
previously  reviewed  and  funded  research  might  not  be  subject  to  peer 
review.  The  Environmental  Protection  Agency’s  peer  review  guidance 
recognizes  circumstances  that  might  preclude  peer  review,  including 
research  methodology  that  is  widely  accepted  and  research  for  which  the 
regulatory  activity  has  been  terminated. 


Agency  Comments 


We  provided  a  draft  of  this  report  to  the  Agricultural  Research  Service,  the 
Cooperative  State  Research,  Education,  and  Extension  Service,  and  the 
Forest  Service  within  the  U.S.  Department  of  Agriculture;  the  Department 
of  Commerce;  the  Department  of  Energy;  the  Environmental  Protection 
Agency;  the  National  Institutes  of  Health;  the  Department  of  the  Interior; 
the  National  Aeronautics  and  Space  Administration;  the  National  Science 
Foundation;  the  Department  of  Transportation;  and  the  Office  of  Science 
and  Technology  Policy  for  review  and  comment.  We  obtained  comments 
from  each  of  the  above  agencies.  Generally,  the  agencies  concurred  that 
the  report  provided  an  accurate  portrayal  of  their  peer  review  practices. 
Some  of  the  agencies  suggested  technical  changes  to  the  report  to  help 
ensure  an  accurate  description  of  their  peer  review  practices,  and  we 
incorporated  the  agencies’  comments.  An  official  at  the  Department  of 
Energy  pointed  out  that  our  statement  that  peer  review  practices  vary  is 
not  without  policy  connotations  and  is  subject  to  misinterpretation.  He 
said  that  peer  review  practices  should  vary  among  and  within  the 
agencies.  To  address  this  concern,  we  added  additional  agency  views  on 
the  need  for  flexibility  in  the  peer  review  process. 
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Scope  and 
Methodology 


To  define  what  is  meant  by  peer  review  and  to  describe  the  federal 
government’s  peer  review  policy,  we  reviewed  studies  of  government  peer 
review,  previous  gao  reports,  and  documentation  provided  by  the  12 
agencies  included  in  our  review.  We  also  interviewed  officials  from  the 
Office  of  Science  and  Technology  Policy  and  reviewed  that  office’s 
guidance  and  its  and  the  Office  of  Management  and  Budget’s  annual 
budget  direction  to  federal  agencies. 

To  describe  the  peer  review  practices  of  12  agencies,  we  obtained  and 
compared  descriptive  information  on  peer  review  at  each  agency  to 
identify  what  the  various  practices  were  and  to  determine  whether  the 
practices  were  uniform  among  and  within  the  agencies.  The  agencies 
provided  legislation,  policies,  manuals,  and  other  documentation,  which 
we  reviewed,  related  to  the  agencies’  implementation  of  peer  review.  The 
agencies  also  provided  fiscal  year  1999  research  and  development  budget 
data.  To  obtain  the  agencies’  rationales  for  their  practices  and  the  reasons 
for  the  variations  among  the  agencies  and  programs  and  to  obtain 
information  about  practices  that  were  not  formally  documented,  we 
interviewed  officials  knowledgeable  about  and  responsible  for  conducting 
peer  reviews  of  scientific  research  at  each  agency’s  Headquarters 
headquarters  office. 

To  identify  other  quality  assurance  reviews  the  agencies  conducted  in 
addition  to  or  in  lieu  of  peer  reviews,  we  reviewed  agency  documentation 
and  interviewed  agency  officials.  Because  the  12  agencies’  practices  were 
carried  out  at  numerous  research  sites  at  headquarters,  field  offices, 
laboratories,  research  stations,  and  grantee  locations  across  the  United 
States,  we  did  not  attempt  to  verify  the  extent  to  which  the  many  different 
practices  reported  by  the  agencies  were  being  implemented  and  carried 
out. 

From  our  interviews  with  agency  officials,  we  also  identified  research  that 
did  not  receive  any  peer  review  or  other  quality  assurance  review  and  the 
agencies’  rationale  for  not  conducting  reviews  of  this  research.  Our  work 
was  performed  from  August  1998  through  March  1999  in  accordance  with 
generally  accepted  government  auditing  standards. 


As  arranged  with  your  offices,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  10  days  after  its 
date.  At  that  time,  we  will  send  copies  of  this  report  to  Dan  Glickman, 
Secretary,  Department  of  Agriculture;  William  M.  Daley,  Secretary, 
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Department  of  Commerce;  Bill  Richardson,  Secretary,  Department  of 
Energy;  Donna  E.  Shalala,  Secretary,  Department  of  Health  and  Human 
|  Services, ;  Bruce  Babbitt,  Secretary,  Department  of  the  Interior;  Rodney  E. 
Slater,  Secretary,  Department  of  Transportation;  D.  James  Baker,  Under 
Secretary,  National  Oceanic  and  Atmospheric  Administration;  Floyd  P. 
Horn,  Administrator,  Agricultural  Research  Service;  Colien  Hefferan, 
Acting  Administrator,  Cooperative  State  Research,  Education,  and 
Extension  Service;  Mike  Dombeck,  Chief,  Forest  Service;  Daniel  S.  Goldin, 
Administrator,  National  Aeronautics  and  Space  Administration;  Carol  M. 
Browner,  Administrator,  Environmental  Protection  Agency;  Jane  F. 
Garvey,  Administrator,  Federal  Aviation  Administration;  Raymond  G. 
Rammer,  Director,  National  Institute  of  Standards  and  Technology;  Harold 
E.  Varmus,  Director,  National  Institutes  of  Health;  Rita  R.  Colwell, 
Director,  National  Science  Foundation;  Jacob  J.  Lew,  Director,  Office  of 
Management  and  Budget;  Neal  Lane,  Director,  Office  of  Science  and 
Technology  Policy;  and  Thomas  Casadevall,  Director,  U.S.  Geological 
Survey.  We  will  also  make  copies  available  to  others  on  request. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please  call 
me  at  202-512-3841.  Major  contributors  to  this  report  are  listed  in  appendix 
XIII. 


Susan  Kladiva 

Associate  Director,  Energy,  Resources, 
and  Science  Issues 
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Abbreviations 

ars  Agricultural  Research  Service 

csrees  Cooperative  State  Research,  Education,  and  Extension 

Service 

doe  Department  of  Energy 

epa  Environmental  Protection  Agency 

faa  Federal  Aviation  Administration 

fs  Forest  Service 

gao  U.S.  General  Accounting  Office 

nasa  National  Aeronautics  and  Space  Administration 

nih  National  Institutes  of  Health 

nist  National  Institute  of  Standards  and  Technology 

noaa  National  Oceanic  and  Atmospheric  Administration 

nsf  National  Science  Foundation 

ostp  Office  of  Science  and  Technology  Policy 

r&d  research  and  development 

usgs  U.S.  Geological  Survey 
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Peer  Review  Practices  at  the  Agricultural 
Research  Service 


The  following  presents  a  description  of  the  Agricultural  Research  Services’ 
(ars’)  peer  review  and  other  quality  assurance  review  practices. 

The  Agricultural  Research  Service,  an  agency  of  the  U.S.  Department  of 
Agriculture,  conducts  research  on  foods,  fibers,  soil,  water,  and  other 
natural  resources,  ars’  mission  is  to  solve  technical  agricultural  problems 
of  broad  scope  and  high  national  priority,  ars  carries  out  this  mission 
through  23  national  programs,  ars  does  not  have  a  regulatory  mission  but 
develops  methods  and  technologies  used  by  other  regulatory  agencies 
within  the  Department,  such  as  the  Animal  and  Plant  Health  Inspection 
Service  and  the  Food  Safety  Inspection  Service,  and  by  other  federal 
agencies  such  as  the  Food  and  Drug  Administration  and  the 
Environmental  Protection  Agency,  ars’  funding  for  fiscal  year  1999  is 
about  $813  million.  Most  research  projects  are  conducted  in-house  by  ars 
scientists  and  between  ars  scientists  and  states,  local  governments,  private 
firms,  and  institutions  through  cooperative  research  and  other  types  of 
agreements.  Only  about  $4  million  of  ars’  annual  appropriation  is 
competitively  awarded  to  external  scientists.  According  to  agency 
officials,  ars  spends  about  $800,000  annually  to  conduct  peer  reviews. 


Peer  Review 
Definition 


ars  defines  peer  review  as  the  evaluation  of  the  conceptual,  relevance,  and 
technical  soundness  of  research  by  highly  qualified  scientists  active  in  the 
same  or  closely  related  research  fields. 


p  p  •  ^  pp_  ars  has  a  multilayered  system  of  complimentary  peer  reviews  that  includes 

xtJcl  xvOVltJW  He iCl  Ic  reviews  of  the  technical  merit  of  planned  research  projects  prior  to 

funding,  reviews  of  research  results  prior  to  the  publication  of  the  results, 
and  reviews  of  ongoing  research  programs.  Various  ars  directives  and 
manuals  provide  the  guidance  for  conducting  these  peer  reviews,  which 
are  performed  by  both  internal  and  external  experts  and  external 
customers  and  stakeholders,  ars  is  revising  its  project  merit  review  system 
during  fiscal  year  1999  to  satisfy  the  additional  requirements  of  the 
Agricultural  Research,  Extension,  and  Education  Reform  Act  of  1998, 
which  requires  merit  review  panels  composed  of  a  majority  of  external 
peers  for  all  ars  research  projects. 


Reviews  of  Research 
Proposals 


ars’  Research  Project  Documentation  Manual  provides  the  guidance  for 
conducting  the  project  plan  merit  reviews  prior  to  deciding  which  projects 
to  fund.  The  agency  manages  1,100  on-going  research  projects,  which 
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Reviews  of  In-Progress 
Research 


operate  at  102  locations  and  are  assigned  to  specific  laboratories  based  on 
program  plans.  The  projects  span  3  to  5  years,  and  each  year  200  to  300 
projects  require  funding  or  reduction  decisions,  ars’  internal  peers,  who 
are  not  involved  in  the  funding  decision,  review  detailed  project  plans.  In 
addition,  the  list  of  peers  must  include  three  individuals  from  universities 
and  three  individuals  from  customer  or  stakeholder  groups.  Each  reviewer 
provides  comments  by  mail,  ars  managers  request  the  reviews  and  act  on 
the  results.  The  ars  scientist  who  prepared  the  project  plan  must  respond 
in  writing  to  reviewers’  comments,  ars  officials  said  that  the  new  project 
merit  review  system  will  consist  of  peer  review  panels  that  will  review  the 
merit  of  research  project  proposals  related  to  each  of  the  23  national 
programs.  To  ensure  that  the  new  system  meets  the  letter  and  spirit  of  the 
law,  ars  officials  said  that  they  will  draw  on  the  expertise  of  other  science 
agencies,  such  as  the  National  Science  Foundation,  the  National  Academy 
of  Sciences,  and  the  Cooperative  State  Research,  Education,  and 
Extension  Service. 

ars’  national  program  staff  manages  the  peer  review  process.  Currently, 
peers  are  selected  by  ars  area  directors  in  consultation  with  both  in-house 
and  external  experts  in  relevant  fields.  Under  the  revised  system,  the 
national  program  staff  will  select  peer  reviewers  from  candidate  pools. 

The  ars  scientist  whose  plan  is  reviewed  suggests  names  of  scientific 
peers  for  consideration.  Criteria  for  all  reviewer  selections  include  a  high 
level  of  subject  matter  knowledge  and  accomplishment  in  scientific 
research.  According  to  ars  officials,  the  majority  of  reviewers  come  from 
outside  ars,  and  the  continuous  involvement  of  customers,  stakeholders, 
and  ars  managers  helps  ensure  high-quality  research,  ars  reviews  of 
proposals  and  all  scientific  research  are  based  on  six  assessment  criteria: 
overall  scientific  value;  probability  of  success;  adequacy  of  approach  and 
excellence  of  research  procedure;  adequacy  of  literature  review  and  the 
researcher’s  knowledge;  extent  of  duplication  of  other  research;  and 
reviewers’  suggestions  for  improvement. 

As  congressionally  mandated,  about  $4  million  of  ars’  annual 
appropriation  is  competitively  awarded  through  grants  to  external 
scientists  to  conduct  research  in  specific  program  areas.  These  proposals 
|  are  peer-  reviewed  by  a  mix  of  internal  and  external  peer  reviewers. 


Periodically,  ars  area  offices  organize  in-depth  location  reviews.  These 
reviews  are  conducted  by  panels,  usually  in  a  workshop  format  with  panel 
members  from  ars,  other  agencies,  academia,  and  the  private  sector  and 
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generally  require  several  days  to  complete.  Location  reviews  may  evaluate 
the  quality  and  impact  of  research  performed  on  several  projects.  These 
reviews  have  multiple  objectives  and  provide  input  to  a  variety  of 
management  issues  such  as  personnel,  productivity,  quality,  and  financial 
management.  In  addition,  ars  forms  ad  hoc  panels  to  address  more 
immediate  research  issues  as  needed.  These  panels  gather  information  on 
specific  problems  and  identify  possible  corrective  actions. 

Reviews  of  Publications 

ars  requires  its  scientists  to  report  research  results,  generally  through 
peer-reviewed  scientific  journals,  ars  research  leaders  may  require  that 
research  manuscripts  be  reviewed  by  two  or  more  internal  or  external 
peers  prior  to  submission  to  a  scientific  journal.  These  reviews  are 
obtained  by  mail,  and  the  authors  must  address  reviewers’  comments 
before  the  manuscript  is  approved  for  publication.  According  to  ars 
officials,  publication  in  a  peer-reviewed  journal  is  a  means  to  ensure  the 
quality  and  productivity  of  the  research. 

Other  Peer  Reviews 

ars  also  uses  panels  of  in-house  scientific  peers  to  evaluate  scientists’ 
research  accomplishments  every  3  to  5  years.  The  panels  anonymously 
evaluate  the  scientist’s  research  contributions  by  documenting  the  impact 
of  the  research  using  peer  and  stakeholder  input.  The  results  of  these 
reviews  determine  a  scientist’  grade  promotion.  An  ars  directive 
documents  the  panel  procedures,  including  the  assurance  of  diversity  in 
panel  selection. 

Other  Agency  Quality 
Assurance  Reviews 

Supervisors  and  line  managers  also  conduct  reviews  of  research.  National 
program  staff  research  managers  conduct  internal  project  level  reviews  to 
help  determine  the  impact  of  the  research,  correct  problems,  and  plan 
future  direction. 

Reviews  of  In-Progress 
Research 

ars  lead  scientists  prepare  annual  project  progress  reports,  which 
document  accomplishments,  published  manuscripts,  and  patents.  The 
reports  are  reviewed  by  line  managers  and  submitted  to  the  national 
program  staff,  and  are  used  to  adjust  or  revise  program  plans,  ars  is  now 
introducing  a  revised  system,  which  will  include  an  annual  summary 
progress  report  highlighting  major  research  accomplishments  for  each  of 
the  23  national  programs,  ars  plans  to  place  these  reports  on  the  Internet 
and  to  solicit  comments  from  customers,  stakeholders,  and  scientific 
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peers.  The  national  program  staff  reportedly  spends  about  one-third  to 
one-half  of  their  time  conducting  reviews  of  projects  and  programs  in  the 

23  major  program  areas,  ars  officials  said  that  the  individual  scientist’s 
annual  performance  appraisal  is  also  a  basic  tool  for  assessing  scientists’ 
accountability  for  the  accomplishment  of  research  project  objectives. 

Other  Reviews 

ars’  program  planning  process  includes  setting  priorities  and  ensuring  that 
research  is  relevant  to  the  agency’s  mission.  Internal  sources  and  external 
customers  and  stakeholders,  including  the  Congress,  policymakers, 
consumer  groups,  private  industry,  and  academia,  provide  input  to  the 
planning  process  through  national  program  planning  workshops.  These 
reviews  help  with  the  development  of  the  agency’s  strategic  plan  and  the 
national  programs,  ars  also  plans  to  utilize  the  National  Agricultural 
Research,  Extension,  Education,  and  Economics  Advisory  Board  to 
oversee  the  implementation  of  the  peer  review  system  and  ars’ 
priority-setting  process. 

Research  Not  Subject 
to  Review 

The  Congress  directs  ars  to  allocate  about  $10  million  annually  to  external 
researchers  for  specific  cooperative  agreements,  ars  usually  does  not 
conduct  peer  review  of  this  research  prior  to  awarding  the  funds. 

However,  outside  researchers  must  provide  annual  progress  reports  that 
are  reviewed  by  ars’  managers. 
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The  following  presents  a  description  of  the  Cooperative  State  Research, 
Education,  and  Extension  Service’s  (csrees’)  peer  review  and  other  quality 
assurance  review  practices. 

The  Cooperative  State  Research,  Education,  and  Extension  Service,  within 
the  U.S.  Department  of  Agriculture,  was  created  in  1995  through  a  merger 
of  the  Cooperative  State  Research  Service  and  the  Extension  Service. 
csrees  funds  research  to  address  problems  of  national  and  regional 
importance  to  agriculture,  forestry,  and  related  sciences.  The  agency’s 
fiscal  year  1999  research  budget  is  about  $465  million.  About  $237  million 
is  designated  for  noncompetitive  formula  funding  land  grant  universities. 
The  remainder  of  the  budget  funds  competitive  grants  ($150  million)  and 
congressionally  mandated  special  research  grants  ($78  million),  csrees  has 
no  regulatory  role,  but  its  research  can  be  used  by  regulatory  agencies.  For 
example,  the  Environmental  Protection  Agency  may  use  results  from 
csrees’  Pesticide  Management  Alternatives  Program  to  determine  if 
changes  should  be  made  to  regulations  on  pesticide  use. 


Peer  Review 
Definition 


csrees  does  not  have  a  formal  definition  of  peer  review.  However,  the 
agency  is  developing  definitions  of  both  peer  review  and  merit  review  as 
part  of  an  agency  initiative  studying  how  the  agency  will  comply  with  the 
Agriculture’s  new  legislative  mandates  related  to  peer  review.  Section  103 
of  the  Agricultural  Research,  Extension,  and  Education  Reform  Act  of 
1998  (the  act)  requires  “peer  review”  of  all  csrees  research  grants  issued 
on  a  competitive  basis  and  “merit  review”  of  competitive  extension  and 
education  grants  but  does  not  define  these  terms.  A  csrees  official  said 
that  merit  review  would  probably  be  defined  as  an  evaluation  of  a  project 
or  program  to  determine  its  technical  quality  and  relevance  to  program 
goals.  Peer  review  will  probably  be  defined  as  a  method  for  conducting 
merit  review  that  uses  people  with  qualifications  and  expertise  to  conduct 
research  similar  to  that  being  reviewed. 


p  -p  ■  p  r*ti  cpc  csrees  funds  both  competitive  and  noncompetitive  research,  but  generally 
r  Ivfci  v  external  peer  review  applies  only  when  entities  compete  for  funding.  The 

|  act  requires  that  all  research  funded  by  Agriculturethe  Department  be 
peer-reviewed  and  that  a  peer  review  panel  assess  the  merit  and  relevance 
of  the  research  at  least  every  5  years.  A  csrees  official  said  that  flexibility 
will  be  critical  in  developing  methods  used  to  select  reviewers  and 
conduct  reviews,  depending  on  the  types  of  decisions  required  and  the 
field  of  science  involved.  For  example,  a  goal  of  the  Pesticide  Alternatives 
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Program  is  to  determine  the  most  important  pesticides  and  crops  to 
investigate.  Therefore,  two  peer  review  panels  were  convened  to  review 
proposals:  one  with  experts  in  the  related  scientific  field  and  another  with 
experts  in  policy  and  in  research  and  funding  priorities. 


Reviews  of  Research 
Proposals 


All  programs  that  award  competitive  grants  use  formal  documented  peer 
review  methods  and  select  external  peers.  The  $100  million  National 
Research  Initiative  Competitive  Grants  Program,  which  funds 
high-priority,  fundamental,  and  mission-linked  research  revised  its  peer 
review  rules  in  1994.  A  program  description  that  includes  peer 
review-related  instructions  is  now  published  annually.  The  competitive 
grant  application  process  is  open  to  anyone  in  the  United  States,  including 
other  Agriculture  scientists.  The  requirement  to  conduct  peer  reviews  of 
research  grant  applications  is  found  in  7  C.F.R.  part  3411,  which  specifies 
the  requirements  for  composition  of  peer  review  groups  and  ad  hoc 
reviewers,  including  the  need  to  include  experts  from  universities, 
industry,  and  private  consultants  and  from  a  variety  of  locations.  This 
regulation  also  lists  detailed  evaluation  criteria  in  three  categories: 
scientific  merit;  qualifications  of  proposed  project  personnel  and  adequacy 
of  facilities;  and  relevance  to  long-range  improvements  in,  and 
sustainability  of,  U.  S.  agriculture. 

In  fiscal  year  1998,  the  National  Research  Initiative  Competitive  Grants 
Program,  cskees’  largest  competitive  grant  program,  with  an  appropriation 
of  $98  million,  spent  $4.3  million  for  its  peer  reviews  of  competitive 
research  proposals.  An  agency  official  said  that  peer  review  is  expensive 
because  of  the  cost  of  the  panel  process.  The  agency  pays  panel  managers 
on  an  hourly  basis  for  up  to  40  days  of  work  plus  travel  expenses;  panel 
members  receive  $150  a  day  honorarium  plus  travel  expenses,  csrees  has 
legislative  authority  to  use  program  rather  than  administrative  funds  to 
pay  for  peer  review  panel  costs. 

In  the  case  of  the  National  Research  Initiative  Competitive  Grants 
Program,  specific  peer  review  procedures  differ  among  the  27  research 
areas.  Generally,  panels  of  outside  experts  conduct  reviews,  but  mail  and 
ad  hoc  reviewers  are  also  used  when  additional  scientists  are  needed  or 
the  subject  is  specialized.  A  rotating  position  of  Panel  Manager  chairs  the 
review  panel  for  each  program.  These  positions  are  generally  filled  by 
outside  experts  who  work  for  approximately  3  weeks  a  year.  Each  panel 
member  reviews  about  30  proposals  and  provides  written  comments  for 
about  20.  Three  written  reviews  are  required  for  any  project  recommended 
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for  award.  According  to  a  departmental  report  on  the  quality  of  research,8 
csrees  scientists  select  panel  members  and  provide  leadership  by 
overseeing  the  review  process.  The  scientists  may  share  administrative 
responsibilities  with  panel  managers,  serve  as  panel  Chairs  if  there  is  no 
outside  panel  manager,  and  are  to  ensure  adherence  to  documented 
conflict  of  interest  and  confidentiality  rules.  Most  peer  review  panels  are 
exempt  from  the  Federal  Advisory  Committee  Act.9  For  this  reason,  the 
panels  can  reach  consensus  on  their  recommendations  for  funding 
proposals  while  maintaining  confidentiality.  According  to  a  csrees  official, 
the  selecting  officials  accept  the  panels’  recommendations  with  few 
exceptions,  csrees  funds  about  20  percent  of  the  applications  it  receives.  A 
summary  and  copies  of  written  reviews  are  sent  to  the  researcher,  and 
only  names  of  awardees  are  made  public.  There  is  no  appeals  process  for 
applicants  that  are  turned  down,  but  feedback  is  provided,  and  applicants 
|  can  reapply  the  following  year.  Section  103  of  the  aAct  requires  colleges  to 
peer  review  all  research  as  criteria  for  submitting  grant  proposals. 
However,  a  csrees  official  said  that  although  many  colleges  peer  review 
their  research,  csrees  will  continue  to  conduct  peer  reviews. 


Reviews  of  In-Progress  When  competitively  funded  research  projects  are  scheduled  for  renewal 

Research  either  annually  or  every  2  to  3  years,  progress  reports  are  peer-reviewed, 

and  the  projects  compete  for  funding  with  new  proposals  under  the 
competitive  grants  merit  review  process.  If  the  review  panel  considers  the 
progress  unsatisfactory,  the  project  will  not  be  funded.  A  csrees  official 
said  that  40  to  50  percent  of  renewals  are  funded  through  this  process. 


Other  Agency  Quality 
Assurance  Reviews 


csrees  internally  reviews  the  noncompetitive  research  proposals  it  funds, 
as  well  as  the  progress  of  research  projects  and  programs. 


Reviews  of  Research 
Proposals 


Authority  to  approve  noncompetitive  research  projects  is  delegated  to 
csrees  scientists.  Academic  institutions  submit  research  project  proposals 
that  are  funded  noncompetitively  through  legislatively  established 
formulas.  To  comply  with  the  requirement  in  the  Agricultural  Research, 
Extension,  and  Education  Reform  Act,  grantees  are  now  required  to 
certify  that  the  proposal  has  received  a  peer  review  at  the  institution  level. 
csrees  scientists  will  continue  to  conduct  a  merit  review  of  the  technical 
quality.  In  addition,  section  212  of  the  act  requires  that  grantees  arrange 


8Quality  of  Agricultural  Research.  Report  of  the  Research,  Education,  and  Economics  Quality  Research 
Initiative  Task  Force  (Jan.  1996). 

9P.L.  104-124,  Sec.  803,  (Apr.  4,  1996). 
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for  peer  review  of  research  funded  through  congressionally  mandated 
special  grants,  csrees  scientists  review  the  technical  merit  of  the  special 
grant  proposals  and  may  obtain  views  from  nationally  recognized  experts 
in  the  research  area.  If  a  proposal  is  found  lacking,  csrees  staff  work  with 
the  author  to  improve  the  proposal. 

Department  heads  and  experiment  station  directors,  within  the  originating 
region  of  the  proposal,  review  proposals  for  regional  research  projects, 
which  involve  collaborative  efforts  of  researchers  at  several  universities 
and  laboratories.  Criteria  for  the  selection  of  regional  projects  are: 
significance  and  scope  of  problem,  level  of  interdependency  and 
cooperation  among  participants,  quality  of  science,  and  the  avoidance  of 
unnecessary  duplication  of  research. 


Reviews  Of  In-Progress  According  to  Agriculture’s  report,  department  or  program  reviews  are 

Research  conducted  internally  at  the  request  of  land-grant  institutions,  with  some 

outside  scientists  conducting  evaluations  of  programs.  Site  visits  are  made 
when  possible  to  laboratories  and  regional  project  locations.  Annual 
j  progress  reports,  which  are  required  by  csrees’  “Special  Terms  and 
Conditions”  for  research  grants,  are  reviewed  internally.  All  grantees  are 
required  to  provide  a  final  technical  report  for  review  when  the  grant 
expires. 


Other  Reviews  While  csrees  has  not  had  a  formal  systematic  process  for  evaluating 

research  performance,  other  than  through  annual  progress  reports,  a 
csrees  official  said  that  the  agency  is  reporting  the  performance  and 
outcomes  of  its  research,  as  required  by  the  Results  Act.  One  reporting 
concern  is  that  research  may  not  always  have  a  positive  outcome,  as 
negative  outcomes  are  part  of  good  science  and  the  learning  process.  The 
departmental  report  states  that  potentially,  the  performance  report 
required  by  the  Results  Act  will  provide  a  mechanism  to  more 
systematically  monitor  and  report  research  performance  and 
accomplishments.  University  partners  provide  information  on  research 
performance  and  outcomes  that  csrees  uses  in  developing  its  performance 
report. 

Consensus  regarding  national  csrees  funding  priorities  is  developed 
through  input  from  a  wide  array  of  private  and  public  sector  stakeholders, 
including  various  commodity  groups,  and  associations.  In  addition,  csrees 
receives  advice  from  the  National  Agricultural  Research  Extension, 
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Education,  and  Economics  Advisory  Board.  The  national  research 
initiatives  program  receives  advice  and  oversight  from  its  Board  of 
Directors,  which  meets  twice  a  year.  Peer  review  panels  also  evaluate 
programs  and  make  suggestions  for  revising  the  annual  call  for  grant 
proposals. 


Research  Not  Subject 
to  Review 


All  research,  either  at  the  project  or  program  level,  is  peer  reviewed  by 
csrees  or  grantee  institutions. 
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The  following  presents  a  description  of  the  Forest  Service’s  (fs’)  peer 
review  and  other  quality  assurance  review  practices. 

The  Forest  Service,  within  the  U.S.  Department  of  Agriculture,  has  two 
primary  authorities  for  conducting  research.  The  first,  the  Forest  and 
Rangeland  Renewable  Resources  Planning  Act  of  1974,  which  authorizes 
fs  to  conduct  natural  resource  assessments,  provide  periodic  reviews  of 
its  research  activities,  and  to  project  a  research  program  every  5  years.10 
The  second,  the  Forest  and  Rangeland  Renewable  Resources  Research  Act 
of  1978,  which  authorizes  the  forestry  research  program.11  In  response,  fs 
conducts  primarily  intramural  research  through  seven  Forestry  Research 
Stations.  The  stations  manage  a  total  of  550  scientists  in  163  Research 
Work  Units  located  at  various  sites  across  the  nation,  including  land  grant 
colleges.  The  agency’s  fiscal  year  1999  r&d  budget  is  $197  million. 
Approximately  10  percent  of  the  budget  will  fund  extramural  research, 
which  generally  is  performed  through  standard  research  agreements  with 
outside  research  partners  at  academic  institutions  or  companies,  or 
cooperative  agreements  with  individual  consultants.  Under  these 
agreements,  which  usually  span  lor  2  years,  fs  scientists  are  substantially 
involved  in  conducting  the  research.  Peer  reviews  are  considered  to  be 
essential  research  activities,  and  fs  does  not  track  the  costs  of  conducting 
the  reviews,  fs  scientists  spend  about  10  percent  of  their  time  serving  as 
peer  reviewers  for  fs  research,  external  journals,  and  other  agencies, 
including  the  National  Science  Foundation. 


Peer  Review 
Definition 


fs  defines  peer  review  as  technical  or  scientific  review  and  defines  peers 
as  scientists  who  have  scientific  expertise  in  and  knowledge  of  the 
discipline  that  is  being  reviewed.  Peers  include  researchers  and 
administrators  from  other  research  stations,  fs  headquarters  staff 
scientists,  university  researchers  and  research  administrators,  and  interest 
group  scientists.  Reviewers  must  be  independent  of  the  research  under 
review.  They  are  selected  or  invited  to  participate  in  reviews  for  which 
they  are  deemed  best  qualified,  based  on  their  scientific  reputation, 
standing  in  their  discipline,  and  specific  knowledge  of  the  research  area. 
Peers  are  sometimes  selected  to  provide  training  or  personal  growth 
opportunities  in  different  programs  and  issues.  Particular  attention  is  paid 
to  obtaining  a  balanced  set  of  viewpoints  and  ensuring  gender  and  ethnic 
diversity,  and  research  administrators  are  responsible  for  ensuring  that  a 
wide  range  of  peers  are  used. 


i0P.L.  93-378,  (Aug.  17, 1974). 
UP.L.  95-307,  (June  30, 1978). 
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p  p p^  pw  pra rps  Although  peer  review  is  not  explicitly  required  by  statute,  agency  officials 

r  eei  he Vie  W  ridtuteo  said  that  peer  review  requirements  documented  in  fs’  manual  satisfy  the 

law’s  requirement  that  fs  encourage  cooperators  and  grantees  to  use  the 
best  available  scientific  skills  from  a  variety  of  disciplines  in  and  outside 
the  fields  of  agriculture  and  forestry,  fs  officials  said  that  peer  review  is 
the  foundation  of  scientific  credibility  and  reputation  and  the  cornerstone 
of  its  quality  assurance  and  control  efforts.  Therefore,  all  research  is  peer 
reviewed.  The  peer  reviews  occur  at  six  different  points  in  the  research 
process:  preparation  of  when  research  work  unit  descriptions  are 
prepared;  when  preparation  of  problem  analyses  are  prepared;  when 
preparation  of  study  plans  are  prepared;  when  the  Deputy  Chiefs  makes 
his  or  her  program  reviews  and  the  Chiefs  overviews  of  research  stations; 
when  preparation  of  research  publications,  articles,  and  other  research 
products  are  prepared;  and  when  scientists’  position  descriptions  are 
evaluated  Research  Grade  Evaluation  Guide  evaluations  of  scientist 
position  descriptions. 


Reviews  of  Research 
Proposals 


To  help  determine  research  priorities  and  direction,  and  which  research  to 
perform,  fs  research  work  unit  descriptions,  problem  analyses,  and  study 
plans  are  subject  to  peer  review.  A  research  work  unit  description  is  a 
concise  summary  of  the  work  unit’s  mission,  problems  to  be  solved 
through  research,  reason  the  problems  were  selected,  proposed  research 
approach,  planned  accomplishments,  and  staffing  needs.  The  criteria  for 
setting  research  priorities  or  selecting  specific  problems  for  study  are 
subjective  and  depend  on  the  nature  of  the  decision  being  made.  Peer 
reviews  lead  to  decisions  about  prioritizing  problems  and  general 
approaches  to  solving  them,  level  of  resources  needed,  accomplishments, 
and  likely  benefits.  The  descriptions  are  revised  about  every  5  years  and 
are  finalized  after  several  iterations  of  review  and  comment.  Technical 
assistance  visits  to  research  stations  provide  peer  review  of  the  research 
details.  Prior  to  final  approval,  the  descriptions  are  submitted  to  fs 
headquarters  for  another  peer  review.  The  Station  Director  then  approves 
the  description  and  the  Deputy  Chief  for  Research  and  Development 
concurs  with  decisions  contained  in  the  description. 

The  problem  analysis  contains  precise  definitions  of  the  research  problem, 
benefits  to  customers,  and  the  likelihood  that  the  research  will  provide  a 
solution.  The  analysis  is  a  detailed  plan  prepared  by  a  project  leader  or 
designated  scientist  after  approval  of  the  research  work  unit  description. 
The  author  selects  peer  reviewers  based  on  his  or  her  knowledge  of  their 
interests  and  capabilities.  Following  the  peer  review,  an  Assistant  Station 
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Reviews  of  In-Progress 
Research 


Director  for  Research  or  a  Program  Manager  must  approve  the  problem 
analysis. 

Last,  a  study  plan  is  prepared,  usually  by  the  scientist  assigned  to  perform 
the  research.  The  plan  defines  the  research  objectives  and  detailed 
methods  to  be  applied.  The  plan  is  submitted  usually  by  mail,  to  peers  who 
are  selected  by  the  author  or  project  leader.  Following  peer  review,  the 
researcher  modifies  the  study  plan  as  needed  needed,  and  the  Project 
Leader  or  another  research  manager  approves  the  plan.  The  peer  review 
comments  and  researcher  responses  are  documented  with  the  study  plan 
files. 

fs  does  not  have  any  documented  instructions  regarding  conflict  of 
interest  in  peer  reviews  and  considers  this  to  be  an  ethics  issue.  A  fs  task 
group  is  currently  developing  a  policy  on  professional  ethics  to  ensure 
against  misconduct  in  scientific  research,  fs  officials  said  that  the  peer 
review  process  has  built  in  checks  and  balances  to  ensure  independence. 
For  example,  obtaining  a  minimum  of  three  reviews,  or  inviting  10  to  20 
people  to  participate  in  a  research  work  unit  description  review,  ensures 
that  a  diversity  of  views  will  be  obtained.  The  officials  said  that  often 
researchers  seek  out  their  harshest  critics  to  review  their  work  to  get  early 
insight  into  concerns.  Individual  researchers  are  responsible  for  soliciting 
peer  reviews  and  deciding  how  to  respond  to  review  comments.  These 
decisions  ultimately  affect  their  scientific  reputation  and  standing  in  their 
field. 

Less  than  5  percent  (or  about  $1  million)  of  the  extramural  budget  funds 
competitive  and  noncompetitive  research  grants.  For  example,  grants  are 
awarded  to  university  researchers  in  response  to  solicitations  for 
mission-related  research.  The  grant  agreements  include  study  plans  that 
are  subject  to  the  same  review  process  as  is  the  intramural  research 
discussed  above. 


A  study  plan  can  span  many  years,  such  as  a  45-year  study  of  the  impacts 
|  of  tree-  cutting  methods.  Such  long-range  plans  are  reviewed  every  3  to  5 

years.  Program  reviews  and  technical  assistance  visits  help  station  and 
headquarters  managers  reach  agreement  on  research  priorities  and 
direction  and  ensure  the  research  is  still  needed  to  address  priorities. 
Program  reviews  normally  cover  several  research  work  units,  and 
technical  assistance  visits  usually  cover  a  single  unit’s  research.  All 
program  reviews  have  peers,  including  senior  external  customers,  such  as 
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state  foresters  arid  university  deans.  In  addition,  the  Deputy  Chiefs 
program  reviews  and  Chiefs  overview  of  research  stations  evaluate  a 
research  station’s  overall  program  and  are  conducted  on  a  cyclical  basis  to 
improve  the  station’s  overall  research  program  results.  Seven  review 
topics  are  addressed  during  these  reviews,  with  the  objective  of  uniformly 
evaluating  the  research  program  to  ensure  appropriate  quality  and  quantity 
of  results. 


Reviews  Of  Publications  According  to  fs  officials,  publications  play  a  key  role  in  the  quality 

assurance  and  quality  control  process  for  research  and  development.  Peer 
reviews  of  manuscripts  prior  to  publication  are  part  of  the  process  and 
help  ensure  that  the  quality  of  the  research  is  high  and  results  are 
significant  enough  to  warrant  dissemination.  Other  indicators  of  quality 
exist,  such  as  science  citation  abstracts,  book  reviews,  publication  request 
rates,  book  sales  by  vendors,  invitations  to  speak,  and  requests  for 
consultations,  but  these  are  not  substitutes  for  the  peer  review  of 
publications.  Three  peers  generally  review  research  products  prepared  by 
fs  researchers  or  external  research  partners.  Products  include 
manuscripts  for  publication  in  external  journals,  research  papers,  research 
notes,  and  general  technical  reports.  For  extramural  research,  the 
necessity  to  obtain  peer  review  is  part  of  the  cooperative  agreement.  The 
research  partner,  the  fs,  or  both  may  conduct  the  reviews.  For  internal  fs 
publications,  and  publications  in  external  journals  that  do  not  require  peer 
review,  the  names  of  reviewers  are  documented  on  a  manuscript  approval 
form,  and  the  author  must  provide  a  justification  for  any  reviewer’s 
comments  to  which  he  or  she  did  not  respond. 


Other  Peer  Reviews  A  FS  official  said  that  users  of  fs’  research  results  or  members  of 

stakeholder  groups  also  assist  in  performing  peer  reviews  of  various 
programs.  Their  opinions  on  the  utility  and  quality  of  the  results  and 
program  priorities  help  the  agency  fulfill  its  responsibilities  to  improve 
customer  service.  Another  type  of  peer  review  that  helps  ensure  the 
quality  of  fs  research  is  the  evaluation  of  individual  scientist  position 
descriptions  performed  by  panels  of  four  peers,  two  from  the  same  general 
discipline  as  the  position  being  evaluated  and  two  from  a  different 
discipline.  Reviews  include  the  nature  of  the  scientist’s  research 
assignment,  level  of  supervision  received,  originality  and  creativity 
required  by  the  assignment,  and  qualifications  and  scientific  contributions. 
Personnel  specialists  select  the  peers  with  advice  from  research 
administrators. 
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Other  Agency  Quality 
Assurance  Reviews 


fs’  Manual  describes  other  reviews  of  stations’  research  in  addition  to  peer 
review.  Station  Directors  and  Assistant  Directors  for  Research  conduct 
reviews  of  a  unit’s  overall  research  as  necessary,  but  at  least  biennially. 
Informal  reviews  are  conducted  with  research  work  units,  as  needed,  to 
evaluate  individual  scientist’s  programs  and  improve  research  results. 
Under  the  cooperative  agreements,  fs  monitors  the  research  conducted  by 
its  partners  by  comparing  required  progress  reports  with  the  tasks 
outlined  in  the  study  plan.  In  addition,  the  Forest  Research  Advisory 
Committee  to  the  Secretary  of  Agriculture,  a  formal  group  of  university, 
industry,  and  interest  group  representatives,  annually  reviews  the 
President’s  budget  and  offers  advice  on  current  and  future  priorities. 


Research  Not  Subject 
to  Review 


Research  stations  often  waive  the  requirement  that  three  individuals 
review  a  manuscript  prior  to  publication  if  the  journal  has  a  policy  of 
securing  “blind”  peer  reviews  of  drafts.  Peer  reviews  are  also  not 
required  if  the  research  has  previously  been  published  in  a  peer-reviewed 
journal  or  fs  publication. 
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The  following  presents  a  description  of  the  National  Institute  of  Standards 
and  Technology’s  (nist’s)  peer  review  and  other  quality  assurance  review 
practices. 

The  National  Institute  of  Standards  and  Technology  (nist),  founded  as  the 
National  Bureau  of  Standards  in  1901,  is  a  nonregulatory  agency  within  the 
Department  of  Commerce,  nist’s  primary  mission  is  to  promote  U.S. 
economic  growth  by  working  with  industry  to  develop  and  apply 
technology,  measurements,  and  standards,  nist  carries  out  this  mission 
through  four  major  programs:  the  Measurement  and  Standards 
Laboratories,  the  Advanced  Technology  Program,  the  Manufacturing 
Extension  Partnership,  and  the  Malcolm  Baldrige  National  Quality 
Program.  The  following  information  focuses  on  the  Measurement  and 
Standards  Laboratories  and  the  Advanced  Technology  Program,  nist’s 
primary  r&d  programs. 

The  Measurement  and  Standards  Laboratories  in  Gaithersburg,  Maryland, 
and  Boulder,  Colorado,  work  with  industry  and  government  agencies  to 
advance  measurement  science  and  develop  standards,  nist’s  standards  for 
weight,  size,  volume,  and  other  physical  quantities  are  used  to  ensure 
accuracy,  fairness,  and  efficiency  for  more  than  $3  trillion  worth  of  goods 
and  services  annually.  For  example,  the  standards  for  x-rays,  drugs,  and 
DNA  testing  ensure  the  safety  and  efficacy  of  millions  of  medical 
procedures  each  year,  nist  also  develops  new  standards  to  support 
advances  in  semiconductor  electronics  manufacture,  communications, 
and  information  technology. 

In  1988,  nist’s  mission  was  expanded  to  include  new  programs  designed  to 
further  support  U.S.  industry.  One  of  these  programs,  the  Advanced 
Technology  Program,  provides  co-funding  to  companies  to  initiate 
high-risk  research  to  develop  promising  technologies  with  potential  for 
broad  economic  and  technological  benefits  across  society.  For  fiscal  year 
1999,  nist’s  budget  authority  of  $641  million  includes  about  $233  million 
for  e&d  in  the  Measurement  and  Standards  Laboratories  and  about 
$178  million  for  external  r&d  co-funded  by  the  Advanced  Technology 
Program. 


Peer  Review 
Definition 


nist  defines  peer  review  as  an  evaluation  by  at  least  two  internal  or 
external  technical  experts  familiar  with  the  technical  aspects  of  the 
proposals  or  programs  and  able  to  offer  informed  judgments  about  the 
technical  quality  of  proposed  external  research  and  of  internal  research 
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programs.  The  Department  of  Commerce’s  administrative  procedure 
provides  guidance  on  competitive  reviews  for  the  award  of  external 
funding  in  discretionary  grant  programs,  including  research  grants 
co-funded  by  the  Advanced  Technology  Program  and  external  grants 
awarded  by  the  Measurement  and  Standards  Laboratories. 


Pppr  ft  pvi  p w  Pra Mi  ppq  nist  uses  a  mix  of  internal  311(1  external  reviewers  to  guide 

rcei  neview  ridtliLCS  decision-making  in  the  award  of  external  r&d  grants  and  to  evaluate  the 

technical  quality  of  its  internal  research  programs.  The  reviews  are 
tailored  to  the  particular  purpose  and  structure  of  each  program.  The 
Advanced  Technology  Program  uses  a  combination  of  internal  and 
external  reviewers  at  various  stages  to  determine  which  research  project 
proposals  to  fund.  In  addition,  nist  is  developing  an  external  program-level 
peer  review  for  the  evaluation  of  the  technical  quality  of  the  Advanced 
Technology  Program.  In  the  Measurement  and  Standards  Laboratories, 
nist  uses  external  peer  review  to  evaluate  the  technical  quality  of  the 
laboratories’  programs,  nist  also  uses  separate  internal  peer  review 
processes  to  competitively  award  the  small  number  of  external  research 
grants  the  laboratories  fund  each  year  to  supplement  their  internal  r&d 
efforts. 


Reviews  of  Proposed  Using  a  combination  of  internal  and  external  reviewers,  the  Advanced 

Research  Technology  Program  uses  peer  review  to  determine  which  proposed 

research  projects  to  fund,  nist’s  extramural  Advanced  Technology 
Program  provides  multiyear  funding  to  single  companies  and  industry-led 
joint  ventures.  Research  proposals  are  peer  reviewed,  as  specified  in  15 
C.F.R.  295.4.  At  the  beginning  of  each  competition,  a  Source  Evaluation 
Board  is  formed.  Board  members  are  government  employees,  including 
technical  experts  and  specialists  in  business  and  economics  from  nist  and 
other  government  agencies.  After  the  proposals  are  screened  by  the  Board 
and  determined  to  be  complete  and  responsive,  the  Board  arranges  to 
have  the  proposals  reviewed  by  external  technical  and  business  experts. 
Generally,  the  Board  obtains  three  technical  and  three  business  reviews. 
All  board  members  and  reviewers  must  sign  nondisclosure  statements, 
agree  to  protect  proprietary  information,  and  certify  that  they  have  no 
conflict  of  interest.  The  reviewers  score  each  proposal  numerically  and 
also  provide  a  written  critique  for  Board  consideration.  The  proposals 
deemed  by  the  Board  to  be  of  highest  quality  are  designated  semifmalist 
proposals,  and  the  proposers  are  invited  to  nist  for  an  oral  question  and 
answer  session  with  the  Board.  Following  the  oral  reviews,  the  Board 
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ranks  the  proposals  based  on  a  number  of  factors,  including  the  scientific 
and  technical  merit  of  the  research  and  the  potential  for  broad-based 
economic  benefits.  The  Source  Selecting  Official,  a  senior  official  from  the 
Advanced  Technology  Program  named  by  nist  for  each  competition, 
makes  the  final  funding  decisions.  The  selection  officials  can  deviate  from 
the  Source  Evaluation  Board’s  ranking  to  attain  portfolio  balance. 


Reviews  of  In-Progress  NIST  uses  both  internal  and  external  peer  review  to  evaluate  the 

Research  effectiveness  of  its  programs.  Internal  peer  review,  including  program 

reviews  by  upper  management,  is  used  extensively  in  all  programs. 
External  peer  review  is  used  principally  to  evaluate  the  technical  quality  of 
the  Measurement  and  Standards  Laboratories’  programs. 

Each  year  since  1959,  the  National  Research  Council,  an  advisory  group 
within  the  National  Academy  of  Sciences,  has  coordinated  an  external 
peer  review  of  nist  laboratory  programs  by  members  of  industry  and 
academia.  Currently,  about  150  scientists  and  engineers  assess  the 
technical  quality  of  nist’s  laboratory  programs  through  the  Council’s  Board 
on  Assessment  of  nist  Programs.  Panel  members  visit  nist  both 
individually  and  in  groups,  meeting  with  laboratory  management  and  staff 
to  discuss  planned,  on-going,  and  completed  programs  to  determine  the 
technical  quality  and  relevance  of  the  laboratory  programs.  The  evaluation 
process  typically  entails  a  review  of  specific  projects  and  consultation 
with  individual  researchers.  Board  members  use  the  following  criteria: 
technical  merit,  appropriateness  to  nist  missions,  effectiveness,  adequacy 
of  planning,  adequacy  of  human  and  physical  resources,  appropriateness 
of  other  agency  funding,  industrial  impact  of  research,  and  the  integration 
of  the  laboratories  and  nist  missions.  The  Board  annually  issues  a  report 
on  its  assessment.  The  National  Academy  of  Sciences  reviews  the  report 
according  to  procedures  approved  by  its  Report  Review  Committee,  nist 
and  the  Measurement  and  Standards  Laboratories  use  the  findings  and 
recommendations  to  improve  program  quality  and  to  guide 
decision-making.  Subsequent  Board  assessments  and  reports  consider 
how  nist  has  addressed  previous  findings. 


Other  Agency  Quality 
Assurance  Reviews 


nist’s  statutory  advisory  committee,  the  Visiting  Committee  on  Advanced 
Technology,  meets  quarterly  to  review  nist’s  policies,  budget,  organization, 
and  programs.12  The  committee,  composed  of  members  of  industry, 
academia,  and  government,  is  appointed  by  the  Secretary  of  Commerce. 


12I6U.S.C.  278. 


Page  34 


GAO/RCED -99-99  Federal  Research 


Appendix  IV 

Peer  Review  Practices  at  the  National 
Institute  of  Standards  and  Technology 


The  committee  examines  the  programmatic  direction  and  management  of 
all  of  nist’s  programs,  providing  broad  external  review  and  guidance. 
Committee  findings  and  recommendations  are  summarized  each  year  in  an 
annual  report  that  is  submitted  to  the  Secretary  of  Commerce  and 
transmitted  by  the  Secretary  to  the  Congress. 

Agency  officials  also  track  the  results  of  research  efforts  funded  by  the 
Advanced  Technology  Program.  Using  quarterly  progress  reports  and 
annual  meetings  with  the  recipient,  the  assessments  look  at  the  economic 
impact  of  the  research  projects.  Program  officials  monitor  completed 
projects  for  6  years. 


Research  Not  Subject 
to  Review 


nist  occasionally  conducts  congressionally  earmarked  projects  that  are 
not  separately  reviewed.  Such  projects  account  for  a  tiny  fraction  of  nist’s 
program  funds  and  are  reviewed  according  to  the  procedures  of  the 
sponsoring  program.  For  example,  earmarked  projects  in  the 
Measurement  and  Standards  Laboratories  are  reviewed  as  part  of  the 
National  Research  Council’s  Board  of  Assessment  process.  Currently,  nist 
is  funding  only  one  earmarked  grant. 


The  Board  on  Assessment  does  not  review  the  small  amount  of  intramural 
research  conducted  under  Cooperative  Research  and  Development 
Agreements.  These  agreements  cover  joint  research  efforts  in  which  both 
nist  and  a  cooperating  company  provide  staff,  equipment,  facilities,  and/or 
funds  for  a  project  of  mutual  interest.  These  projects  can  involve 
intellectual  property  owned  by  the  private  companies  and  are  protected  by 
law  from  disclosure  by  nist.  The  nonproprietary  nist  contribution  is 
subject  to  the  same  internal  review  as  all  projects  and  is  not  excluded 
from  the  scope  of  the  annual  assessment. 


Page  35 


GAO/RCED-99-99  Federal  Research 


Eview  Practices  at  the  National  Oceanic  and 
Atmospheric  Administration 


The  following  presents  a  description  of  the  National  Oceanic  and 
Atmospheric  Administration’s  (noaa’s)  peer  review  and  other  quality 
assurance  review  practices. 

Established  in  1970,  the  National  Oceanic  and  Atmospheric  Administration 
(noaa)  is  a  bureau  within  the  Department  of  Commerce,  noaa’s  mission  is 
to  describe  and  predict  changes  in  the  Earth’s  environment  and  to 
conserve  and  manage  wisely  the  nation’s  coastal  and  marine  resources. 
noaa’s  research  is  used  to  support  policy  decisions  about  fishery 
management,  disseminate  data  about  the  Earth’s  climate,  and  issue 
weather  reports/warnings. 

noaa  consists  of  five  mqjor  offices:  the  National  Marine  Fisheries  Service, 
the  National  Ocean  Service,  the  Office  of  Oceanic  and  Atmospheric 
Research,  the  National  Weather  Service,  and  the  National  Environmental 
Satellite  Data  and  Information  Service.  The  Office  of  Oceanic  and 
Atmospheric  Research,  the  National  Ocean  Service,  and  the  National 
Marine  Fisheries  Service  do  much  of  the  research  within  noaa.  There  are 
12  environmental  research  laboratories,  11  joint  academic  institutes,  29 
sea  grant  colleges,  and  6  undersea  research  centers  that  conduct  research 
within  the  Office  of  Oceanic  and  Atmospheric  Research.  The  National 
Ocean  Service  sponsors  research  through  its  coastal  ocean  research 
centers  and  focussed  programs  with  other  offices.  The  National  Marine 
Fisheries  Service  conducts  its’  research  through  5  regional  science  centers 
in  about  30  laboratories.  These  laboratories  are  overseen  by  the  National 
Marine  Fisheries  Service  Science’s  Advisory  Board,  composed  of  five5 
Science  Center  Directors  and  the  Director  of  Science  and  Technology.  The 
National  Marine  Fisheries  Service  also  has  extensive  collaborations  with 
academia,  many  through  cooperative  agreements  and  grants,  noaa’s  fiscal 
year  1999  r&d  budget  is  $577  million. 


Definition  of  Peer 
Review 


noaa  defines  peer  review  as  an  organized  and  objective  method  for 
evaluating  proposed,  ongoing,  and  completed  scientific  work,  by 
individuals  and/or  committees  who  have  equal  or  pre-eminent  standing  in 
the  pertinent  field  of  research  and  knowledge  of  the  type  of  work  being 
reviewed. 


p  t>  pvj  pw  pra  p+t  ppc  Virtually  all  of  noaa’s  research  portfolio  is  evaluated  by  some  type  of  peer 
x  eel  xvtJVlc  W  n dCl  tea  review  process.  Peer  review  is  used  at  both  the  project  and  program  levels 

to  certify  the  correctness  of  procedures,  establish  the  plausibility  of 
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Reviews  of  Proposed 
Research 


results,  and  allocate  scarce  resources.  Currently,  noaa  has  no 
comprehensive  peer  review  policy.  Thus,  each  office  has  its  own  policies, 
which  have  accrued  over  time. 


For  the  most  part,  all  external  research  and  internal  research  is  peer 
reviewed;  however,  the  peer  review  methods  differ  among  and  within  the 
offices.  For  example,  the  Office  of  Oceanic  and  Atmospheric  Research,  the 
National  Ocean  Service,  and  the  National  Marine  Fisheries  Service 
conduct  their  peer  reviews  by  either  peer  panel,  mail,  or  a  combination  of 
the  two.  Reviewers  are  selected  from  a  variety  of  sources,  including 
academia,  industry,  government,  the  international  science  community,  and 
from  within  noaa. 

The  Office  of  Oceanic  and  Atmospheric  Research  has  two  ways  that  it  peer 
reviews  competitive  research  proposals.  If  the  Office  puts  out  a  request  for 
proposals  itself,  the  proposals  that  are  submitted  are  peer  reviewed  by  the 
Office  and  then  assigned  to  individual  research  universities  or  institutes 
funded  by  the  Office.  If  the  research  universities  or  institutes  put  out  a 
request  for  proposals,  then  the  individual  university  or  institute  peer 
reviews  the  proposals  submitted.  However,  each  laboratory,  university,  or 
institute  has  its  own  way  of  conducting  peer  reviews  within  the  Office’s 
guidelines.  Research  is  judged  on  rationale,  scientific  merit, 
innovativeness,  the  qualifications  of  the  principal  investigator,  user 
relationships,  relationships  to  noaa’s  priorities,  programmatic  justification, 
linkages,  and  costs. 

In  the  National  Ocean  Service,  competitive  research  proposals  are  peer 
J  reviewed  in  a  two  two-step  process.  First,  the  proposals  are  distributed  to 
knowledgeable  individuals  for  anonymous  review.  Second,  a  review  panel 
|  of  knowledgeable  individuals  are  provided  of  knowledgeable  individuals 

are  provided  the  proposals  and  reviews  for  discussion  and  separate 
ranking  by  each  panel  member.  Proposed  research  is  judged  on  scientific 
rationale,  technical  merit,  qualifications  of  the  researchers,  and  the  cost  of 
the  proposed  work. 

Within  the  National  Marine  Fisheries  Service,  each  proposal’s  peer  review 
method  depends  on  the  products  or  programs  reviewed  and  is  developed 
by  its  sponsoring  organization — a  particular  fisheries  laboratory,  a 
Regional  Fishery  Management  Council,  and/or  the  peer  review  panel  itself. 
The  schedule  or  format  may  also  depend  on  whether  or  not  the  proposal  is 
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legislatively  mandated.  Within  the  Service,  proposals  are  reviewed  by  mail, 
peer  panels,  committees,  and  agency  staff. 

Reviews  of  In-Progress 
Research 

Every  3  to  5  years,  qualified  peers  review  the  research  programs  of 
|  research  universities,  laboratories,  or  institutes  in  the  Office  of  Oceanic 
and  Atmospheric  Research.  This  review  evaluates  the  programmatic 
accomplishments  and  impacts  in  the  context  of  resources  invested.  Each 
set  of  reviewers  visiting  the  site  consists  of  representatives  from  the 
program,  stakeholders,  external  technical  experts,  or  members  of 
established  Office  of  Oceanic  and  Atmospheric  Research  review  panels. 
The  university,  laboratory,  or  institute  reviewed  develops  the  review 
presentation  based  on  Office  of  Oceanic  and  Atmospheric  Research 
criteria. 

Reviews  of  Publication 

Most  of  noaa’s  research  is  published  in  peer-reviewed  journals.  Principal 
scientists  at  each  laboratory  are  evaluated  in  large  part  on  their 
publications  in  peer-reviewed  journals.  The  National  Marine  Fisheries 
Service’s  Scientific  Publications  Office  publishes  two  peer-reviewed 
journals  of  fishery  science  work  performed  by  noaa  and  non-NOAA 
scientists.  Scientific  publications  undergo  an  internal  technical  review  by 
two  management  levels  within  the  Service.  Additional  peer  review  may  be 
requested  during  the  internal  technical  review. 

Other  Peer  Reviews 

noaa  has  recently  established  a  Science  Advisory  Board  that  is  officially 
chartered  in  accordance  with  the  Federal  Advisory  Committee  Act.  The 
15-member  Board  is  composed  of  eminent  scientists,  engineers,  resource 
managers,  and  educators  reflecting  the  full  breadth  of  noaa’s 
responsibilities,  as  well  as  the  ethnic  and  gender  diversity  of  the  United 
States.  The  Board  advises  noaa’s  Administrator  on  long-  and  short-range 
strategies  for  research,  education,  and  the  application  of  science  to 
resource  management.  The  Board  is  currently  considering  its  potential 
role  in  the  oversight  of  noaa’s  various  science  review  panels.  The  Board’s 
operations  and  support  functions  are  within  the  Office  of  the  Chief 
Scientist. 

• 

noaa’s  Office  of  Research  and  Technology  Applications  conducts  the 
technical  review  and  selection  process  for  applicants  to  the  Small 

Business  Innovative  Research  Program.  The  program  was  established, 
among  other  things,  to  foster  participation  by  minority  and  disadvantaged 
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researchers.  A  certain  portion  (2.5  percent  in  fiscal  year  1999)  of  noaa’s 
external  research  and  development  budget  is  set  aside  for  program 
funding.  The  review  is  performed  by  a  variety  of  experts  whose  views  are 
collected  by  mail. 

Scientific  findings  and  conclusions  of  the  National  Marine  Fisheries 
Service  programs  are  sometimes  reviewed  by  opposing  parties  in  court 
when  fisheries  management  decisions  are  litigated. 


Other  Agency  Quality 
Assurance  Reviews 


Since  all  noaa  research  is  evaluated  by  some  type  of  peer  review  process, 
agency  officials  provided  no  examples  of  other  quality  assurance  reviews. 


Research  Not  Subject 
to  Review 


All  noaa  research  is  subjected  to  peer  review  or  other  technical  reviews  by 
noaa  or  its  customers. 
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The  following  presents  a  description  of  the  U.S.  Department  of  Energy’s 
(doe’s)  peer  review  and  other  quality  assurance  review  practices. 

Created  in  1977,  doe’s  mission  is  to  foster  a  secure  and  reliable  energy 
system  that  is  environmentally  and  economically  sustainable,  to  be  a 
responsible  steward  of  the  nation’s  nuclear  weapons,  to  clean  up  its 
facilities,  and  to  support  continued  U.S.  leadership  in  science  and 
technology.  The  agency  conducts  research  and  development  on  a  variety 
of  topics,  including  fossil,  fusion,  and  nuclear  energy  production;  energy 
conservation;  renewable  energy;  biological  and  environmental  research; 
materials  science;  engineering  and  geoscience;  advanced  computing; 
high-energy  and  nuclear  physics;  nuclear  waste  management; 
environmental  remediation;  radiation;  nuclear  stockpile  management; 
nuclear  nonproliferation;  and  the  Human  Genome  Project. 

doe’s  research  can  affect  a  broad  spectrum  of  federal  policies  and 
regulations.  For  example,  doe  generates  federal  energy-efficiency  rules  for 
the  manufacture,  testing,  and  labeling  of  major  home  appliances  and 
certain  commercial  products.  The  Environmental  Protection  Agency’s 
Office  of  Radiation  Protection  and  the  Nuclear  Regulatory  Commission 
have  used  the  results  of  doe’s  research  as  part  of  the  background  used  to 
set  radiation  standards.  In  addition,  agency  research  was  used  to  set 
standards  for  mobile  pollution  sources  and  fuel  regulations  under  the 
Motor  Vehicle  Information  and  Cost  Savings  Act. 

doe’s  research  and  development  budget  for  fiscal  year  1999  is  $7.8  billion. 
Approximately  80  percent  of  the  budget  will  support  research,  research 
facilities,  and  related  activities  within  the  Department  and  its  national 
laboratory  system.  The  remaining  20  percent  will  support  external 
research  conducted  by  industry,  universities,  public  and  private  research 
institutions,  not-for-profit  organizations,  and  research  and  development 
consortia  through  Department-awarded  grants,  cooperative  agreements 
and  contracts,  and  laboratory-awarded  research  subcontracts. 

Because  of  its  diversity,  doe’s  peer  review  practices  are  guided  by  a  variety 
of  laws  and  regulations.  The  Federal  Acquisition  Regulation,  the  doe 
Acquisition  Regulation,  and  the  Competition  in  Contracting  Act  guide  the 
agency’s  peer  review  practices  for  research  and  development  contracts. 
Research  grants  and  cooperative  agreements,  which  are  awarded  through 
a  merit-based  selection  process,  follow  the  Department’s  Financial 
Assistance  Rules,  as  promulgated  in  the  Code  of  Federal  Regulations  10 
C.F.R.  Part  600. 
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doe  has  no  formal  definition  of  peer  review,  but  practices  peer  review  as 
merit  review  with  peer  evaluation — a  formal,  competent,  and  objective 
evaluation  process  using  specified  criteria  and  the  review  and  advice  of 
qualified  peers.  Peers  must  be  technically  competent  in  the  scientific  or 
technical  field  under  review  and  must  be  free  from  conflict  of  interest. 
Peers  may  come  from  any  source,  including  industry,  academia,  private 
and  nongovernmental  institutions,  government  agencies,  and  their 
associated  laboratories. 


p  p  •  p  p+i  ppc  D0E  uses  merit  review  with  peer  evaluation  to  guide  research  direction  and 

rtitil  IvtiVltiW  r  IdLl  o  to  assess  research  progress.  External  research  is  peer-reviewed  in 

conjunction  with  the  preaward  competitive  selection  process.  This 
research  is  also  reviewed  as  part  of  the  award  renewal  process.  Reviews  of 
laboratory  research  occur  at  both  the  laboratory  and  departmental 
oversight  levels.  In  addition,  laboratories,  user  facilities,  and  major 
research  divisions  have  committees  of  outside  experts  that  provide 
periodic  peer  reviews  of  research  relevance  and  quality.  Research  results 
are  also  extensively  published  in  peer-reviewed  journals.  The  methods  for 
conducting  reviews  are  tailored  to  each  situation.  The  following  provides 
examples  of  the  different  peer  review  practices  among  doe’s  programs. 


Peer  Review 
Definition 


Reviews  of  Research 
Proposals 


With  few  exceptions,  merit  review  with  peer  evaluation  guides  doe 
research,  including  that  by  its  research  laboratories.  For  example, 
regulations  governing  the  Financial  Assistance  Program  require  peer 
review  and  competitive  selection.  The  regulations  specify  that  each  grant 
proposal  normally  receive  a  minimum  of  three  reviews  per  proposal  by 
technically  qualified  experts  in  the  proposed  field,  followed  by  a  peer 
review  panel.  Proposals  are  peer-reviewed  for  scientific  excellence.  The 
Office  of  Science  &  Technology,  in  the  Environmental  Management 
Program,  Project  Selection  Reviews,  for  new  research  and  development 
activities,  combine  the  judgments  of  technical  peers  and  potential  users  of 
the  results.  In  addition,  research  subcontracted  by  doe’s  national 
laboratories  to  outside  researchers  is  governed  by  contract  provisions, 
unless  otherwise  justified  through  formal  documentation.  These 
provisions  require  competitive  selection  processes,  including  merit  review 
with  peer  evaluation. 


Peer  review  is  applied  to  the  selection  and  approval  of  most  laboratory 
field  work  proposals.  Field  work  proposals  are  the  means  by  which  the 
laboratories  formally  propose  future  work  and  seek  authorization  for 


Page  4 1 


GAO/RCED-99-99  Federal  Research 


Appendix  VI 

Peer  Review  Practices  at  the  Department  of 
Energy 


expending  research  and  development  funds.  In  the  Office  of  Science,  all 
field  work  proposals  are  required  to  be  peer-reviewed  for  quality  by 
external,  independent  experts.  Each  laboratory  research  program  is 
reviewed  annually.  For  example,  the  Technology  Development  Program  of 
the  Office  of  Environmental  Management  uses  teams  of  subject  matter 
specialists  from  technical,  regulatory,  business,  and  stakeholder 
perspectives.  In  addition,  peer  review  is  used  to  allocate  available  time  and 
to  select  the  experiments  conducted  at  specialized  research  facilities 
located  at  doe’s  laboratories.  Such  facilities  include  accelerators  for  the 
study  of  high-energy  physics  and  the  world’s  most  powerful  computers 
and  lasers. 

At  the  laboratories,  each  director’s  discretionary  research  and 
development  program  and  the  laboratory  field  work  proposals  are 
reviewed.  The  Laboratory  Directed  Research  and  Development  Program 
provides  certain  laboratory  directors  discretionary  funds  (up  to  6  percent 
of  their  laboratory’s  budget)  to  develop  new  scientific  ideas  and 
opportunities  and  to  initiate  new  directions.  The  laboratories  rely  on 
individual  scientific  investigators  and  the  scientific  leadership  of  the 
laboratory  to  identify  opportunities  that  will  contribute  to  scientific  and 
institutional  goals. 


Reviews  Of  In-Progress  Peer  review  is  also  used  in  conjunction  with  the  evaluation  of  on-going 

Research  research.  While  the  substance  of  the  reviews  is  similar,  such  as 

considering  the  quality  and  relevance  of  the  research  and  the  investigator’s 
or  research  group’s  record  of  accomplishment,  the  nature  of  the  reviews 
can  differ.  For  example,  the  Office  of  International  Health  Programs  uses 
independent,  external  review  panels  to  conduct  in-progress  reviews.  The 
Office  of  Science  &  Technology  within  the  Environmental  Management 
Program  conducts  technical  reviews  of  continuing  projects  in  their  third 
year  of  support  or  when  reaching  engineering  demonstration,  or  when 
considered  a  new  start,  through  a  formal  process  externally  managed  by 
the  American  Society  of  Mechanical  Engineers.  The  Society  selects 
reviewers  who  assess  technical  excellence,  relevance,  progress,  and 
productivity.  In  addition,  for  new  environmental-management 
technologies,  mid-year  progress  reviews  are  held  annually  for  each 
program  element,  with  potential  users  assessing  the  applicability  and 
performance  requirements. 
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Reviews  of  Publication  Publication  in  open  literature  constitutes  another  form  of  peer  review. 

Publication  of  original  work  is  considered  essential  at  doe,  and  the 
scientists  it  supports  (both  external  and  internal)  are  continually  evaluated 
by  the  quality  of  their  original  research,  as  indicated,  in  part,  by 
publications  in  archival,  peer-reviewed  journals. 


Other  Peer  Reviews 


Retrospectively,  scientists  who  are  independent  of  the  laboratory  conduct 
reviews  of  laboratory  research  in  conjunction  with  program  reviews  and 
advisory  committee  oversight.  These  reviews  provide  advice  on  the 
quality,  relevance,  and  productivity  of  laboratory-conducted  research.  The 
following  are  three  examples  of  such  reviews. 

•  The  Office  of  Science  regularly  conducts  retrospective  peer  reviews  of 
research  and  development  programs  throughout  the  Department,  which 
includes  an  evaluation  of  a  sampling  of  research  projects.  Individual 

|  programs  also  conduct  reviews. . 

•  The  Office  of  Defense  Programs  uses  an  Inertial  Confinement  Fusion 
Advisory  Committee,  constituted  under  the  Federal  Advisory  Committee 
Act,  which  reports  directly  to  the  Assistant  Secretary  for  Defense 
Programs,  to  assess  program  results.  For  highly  classified  research,  the 
Department  interacts  with  the  Department  of  Defense  for  customer 
feedback  on  program  performance. 

•  The  Office  of  Civilian  Radioactive  Waste  Management  uses  peer  review  to 
help  assess  the  quality  and  validity  of  completed  technical  work  and  to 
ensure  the  quality  of  data  for  use  in  adjudicatory  hearings.  Because  of  the 
U.S.  Nuclear  Regulatory  Commission’s  role  under  the  Nuclear  Waste 
Policy  Act,  the  Commission  has  provided  guidance  on  the  conduct  of  peer 
review.  A  primary  selection  criterion  for  peer  reviewers  is  independence. 
When  there  is  a  potential  or  an  apparent  conflict  of  interest  that  may  bring 
the  independence  of  a  participant  into  question,  a  documented  rationale  is 
included  in  the  peer  review  report. 


Other  Agency  Quality 
Assurance  Reviews 


Many  of  doe’s  energy  technology  development  and  related  research  and 
development  programs  are  deliberately  designed  to  accommodate 
industrial  partners.  In  various  ways,  these  industrial  partners  provide 
opportunities  for  external  merit  review  by  engaging  themselves  as  full 
participants  in  planning,  executing,  and  commercializing  the  research  and 
development.  Such  reviews  extend  beyond  the  peer  review  procedures 
that  characterize  science  programs.  For  most  major  technological 
development  programs,  the  formulation  and  enforcement  of  a 
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comprehensive  Quality  Assurance  Program  is  required.  For  the  Energy 
Efficiency  Program,  quality  control  involves  three  stages:  peer  review  for 
basic  research,  merit  review  for  applied  research,  and  market  review  for 
judging  commercial  application. 

Under  reforms  begun  in  1994,  all  of  the  Department’s  new  contracts  for 
the  management  and  operation  of  its  national  laboratories  require  regular, 
performance-based  merit  reviews  of  the  contractor’s  performance. 
Colleagues,  laboratory  superiors,  and  administrators  at  doe  headquarters 
|  evaluate  the  research  and  development  projects.  The  9  nine  multiprogram 
national  laboratories  also  have  various  industrial  advisory  panels  to  review 
research.  In  addition,  all  research  subcontracted  by  the  laboratories  to 
outside  researchers  is  governed  by  contract  provisions  that  generally 
require  periodic  evaluations  of  the  subcontractor’s  performance. 

Panels  constituted  under  the  Federal  Advisory  Committee  Act  frequently 
advise  doe  program  administrators  on  program  content,  quality,  future 
directions,  and  priorities.  For  example,  the  Office  of  Science  uses  advisory 
committees  for  recommendations  on  basic  energy  sciences,  biological  and 
environmental  research,  high-energy  physics,  nuclear  sciences,  and  fusion 
energy.  Similarly,  the  Office  of  civilian  Radioactive  Waste  Management  has 
standing  advisory  committees  and  just  completed  a  2-year  participatory 
peer  review. 

For  classified  nuclear  weapons  design-related  research,  where  no  broad 
industrial,  university,  or  other  independent  source  of  expertise  exists,  a 
process  of  merit  review  exists  within  doe’s  Defense  Programs  laboratories. 
For  example,  every  5  years,  with  annual  updates,  the  three  Defense 
Programs  laboratories  review  the  nuclear  weapons  in  the  active  stockpile 
through  a  formal  internal  peer  review  “Weapons  Appraisal  Process.”  The 
University  of  California,  the  contractor  that  operates  the  Lawrence 
Livermore  and  Los  Alamos  laboratories,  also  uses  a  President’s  Council 
Panel  on  National  Security  to  assess  the  nuclear  weapons  program.  Each 
of  the  laboratories’  directors  also  appoints  review  committees  for  each  of 
the  laboratories’  divisions,  with  members  coming  almost  exclusively  from 
industry  and  academia  but  sometimes  from  doe  and  its  contractors.  The 
committees  report  to  the  laboratory  directors  with  an  assessment  of  the 
division’s  technical  and  scientific  quality.  The  directors,  in  turn,  file  a 
self-assessment  with  a  review  Council  convened  by  the  President  of  the 
University  of  California.  From  this  process,  the  president  reports  to  doe  on 
the  laboratories’  technical  and  scientific  quality.  Finally,  additional 
reviewing  bodies  such  as  JASON  (a  civilian  science  advisory  group),  the 
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National  Academy  of  Sciences,  the  Nuclear  Weapons  Council,  and  other 
senior  advisory  groups  review  doe’s  Defense  Program’s  research  and 
development  program. 


Research  Not  Subject 
to  Review 


According  to  doe  officials,  most  congressional  mandates  and  earmarks, 
which  designate  projects  and  the  institutions  to  conduct  them,  are  not 
subject  to  the  peer  review  process  in  deference  to  the  congressional 
directives.  However,  once  a  grant  is  funded,  it  is  likely  to  receive  merit 
review  before  being  competitively  renewed,  unless  waived  with  a  written 
determination  by  the  project  administrator.  When  merit  review  is  not 
conducted  before  an  award’s  renewal,  the  award  must  be  considered  to  be 
noncompetitive  and  must  meet  different  selection  requirements.13 
Whenever  the  merit  review  system  is  not  used  for  applications  and 
proposals,  the  Director  of  Grants  and  Contracts  must  obtain  written  prior 
approval  for  a  different  review  procedure.  Very  rarely  are  contracts 
peer-reviewed  when  sole-source  selection  is  used,  but  the  administrator 
making  this  decision  must  justify  this  process.  In  addition,  nonreviewed 
grants  cannot  be  extended  for  more  than  6  years;  periodic  reviews  of  the 
research  results  are  another  check. 


1310  C.F.R.  600.6(c). 
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The  following  presents  a  description  of  the  Environmental  Protection 
Agency’s  (epa’s)  peer  review  and  other  quality  assurance  review  practices. 

The  mission  of  the  Environmental  Protection  Agency,  established  in  1970, 
is  to  protect  human  health  and  to  safeguard  the  natural  environment,  epa 
is  organized  into  10  geographic  regions  and  nine  major  offices: 
Administration  and  Resources  Management;  Air  and  Radiation;  the  Chief 
Financial  Officer;  International  Activities;  Policy;  Prevention,  Pesticides, 
and  Toxic  Substances;  Research  and  Development;  Solid  Waste  and 
Emergency  Response;  and  Water,  epa’s  fiscal  year  1999  operating  budget  is 
about  $547  million,  excluding  state  revolving  funds  for  the  design  and 
construction  of  waste  water  and  drinking  water  systems. 


Peer  Review 
Definition 


Peer  review  is  defined  as  a  documented  critical  review  of  a  specific,  major 
scientific  or  technical  work  product,  conducted  by  qualified,  independent 
individuals,  epa  distinguished  this  peer  review  definition  from  what  it 
classifies  as  “peer  input”  which  is  interaction  during  the  development  of 
an  evolving  agency  work  product,  to  provide  an  open  exchange  of  data, 
insights,  and  ideas.  The  difference  between  peer  review  and  peer  input  is 
the  independence  of  the  peer  reviewers  and  their  level  of  involvement. 
Peer  reviewers  should  have  no  material  stake  in  the  proposal  reviewed 
and  should  have  had  no  substantial  involvement  in  the  development  of  the 
proposal. 


Peer  Review  Policies 
and  Practices 


Generally,  all  research  funded  by  the  Office  of  Research  and  Development 
is  peer  peer-reviewed,  as  legislatively  mandated.14  In  1993,  the 
administrator  issued  a  policy  that  statement  that  major  and  scientifically 
and  technically  based  work  products  related  to  major  agency  decisions 
should  normally  be  peer-reviewed.  In  response  to  one  of  our  1994  reports,16 
each  assistant  and  regional  administrator  developed  a  set  of  standard 
operating  procedures  for  peer  review.  In  June  1994,  the  current 
administrator  reissued  to  a  policy  statement  reaffirming  the  central  role  of 
peer  review  to  ensure  that  epa  policy  decisions  rest  on  sound,  credible 
science  and  data.  After  one  of  our  1996  reports  found  that  epa’s 
implementation  of  these  peer  review  procedures  remained  uneven,  epa’s 


|  1440  C.F.R. 

A6Peer  Review:  EPA  Needs  Implementation  Procedures  and  Additional  Controls  (GAO/RCED-94-89, 
Feb.  22, 1994). 
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Science  Policy  Council  issued  a  peer-review  practices  handbook  for 
agencywide  use  in  February  1998.16 

This  handbook  concentrates  mainly  on  guidelines  for  the  retrospective 
review  of  work  products  used  to  support  epa  decisions.  The  following 
provides  examples  of  the  different  peer  review  practices  among  epa’s 
offices. 


Reviews  of  Research 
Proposals 


The  Office  of  Research  and  Development’s  Science  to  Achieve  Results 
Program  issues  requests  for  research  proposals.  These  proposals  are 
selected  through  a  rigorous  peer  review  process  in  which  panels  of 
independent  researchers  from  relevant  fields  review  all  proposals.  The 
reviewers  use  evaluation  criteria  that  emphasize  the  quality  of  science  as 
well  as  the  responsiveness  to  the  program  request.  Proposals  that  are 
rated  very  good  or  excellent  by  the  panels  are  subjected  to  a 
programmatic  review  within  epa  to  ensure  a  balanced  research  portfolio. 
Office  for  Research  and  Development  scientists  and  Program  or  Regional 
Office  staff  conduct  the  programmatic  reviews,  based  on  their  knowledge 
of  program  priorities  and  how  the  research  proposals  complement  the 
intramural  research  program.  The  reviewers  recommend  proposals  for 
funding  to  the  National  Center  for  Environmental  Research  and  Quality 
Assurance. 

Research  proposals  not  under  the  Science  to  Achieve  Results  Program, 
such  as  congressional  earmarks  and  unsolicited  research  proposals,  are 
also  peer  reviewed  for  technical  merit.  For  epa  contracts  and  other 
assistance  agreements,  the  approach  used  to  peer  review  a  major  scientific 
or  technical  work  is  left  up  to  the  peer  review  leader  and  the  epa’s  decision 
maker  who  base  their  decision  on  the  nature  of  the  topic  and  the  intended 
final  product. 


Reviews  of  In-Progress  The  Office  of  Research  and  Development  has  established  the  Board  of 

Research  Scientific  Counselors,  composed  of  nationally-recognized  scientific  and 

engineering  experts.  The  primary  functions  of  the  board  are  to  evaluate 
office  science  and  engineering  research  programs,  laboratories,  and 
research  management  practices;  recommend  actions  to  improve  their 
quality  and  strengthen  their  relevance  to  the  mission  of  epa;  and  to 
evaluate  and  provide  advice  concerning  the  utilization  of  peer  review 
within  the  office  to  enhance  the  quality  of  science  in  epa. 


ir>Peer  Review:  EPA’s  Implementation  Remains  Uneven  (GAO/RCED-96-236,  Sept  24,  1996). 
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In  addition,  the  Science  Advisory  Board — a  Federal  Advisory  Committee 
Act  committee  with  deliberations  open  to  the  public — functions  as  a 
technical  peer  review  panel,  providing  consultation  and  advisory  reviews 
of  work  products  at  various  points  prior  to  their  completion.  Consisting  of 
10  committees  covering  such  topics  as  health,  radiation,  air  quality  and 
drinking  water,  the  board  draws  scientists,  engineers,  and  economists 
from  academia,  industry,  and  the  environmental  community. 


Reviews  of  Publication  EPA  has  always  encouraged  the  publication  of  its  scientific  and  technical 

material  in  peer-reviewed  literature  as  a  means  of  obtaining  independent, 
external  review  of  its  work  products. 


Other  Peer  Reviews  The  Science  Advisory  Board  often  evaluates  and  reviews  the  technical 

basis  for  a  science  policy  position  adopted  by  epa.  For  example,  the  board 
sometimes  serves  as  a  council  of  peers  in  cases  where  action  is  necessary 
in  order  to  address  emerging  environmental  risk  before  all  the  rigors  of 
scientific  proof  are  met.  The  board  also  reviews  selected  scientific 
documents  that  are  used  as  the  basis  for  environmental  regulations. 

Important,  major  epa  rulemakings,  including  those  determined  as 
“significant”  ($100  million  impact  or  more)  by  the  Office  of  Management 
and  Budget,  lend  themselves  to  extensive  external  peer  review.  Generally, 
more  extensive  and  involved  peer  review  with  external  peers  is  indicated 
for  work  with  more  complex  science,  greater  cost  implications,  or  a  more 
controversial  issue.  Other  projects  that  are  under  strict  time  constraints, 
are  of  lesser  impact  or  less  controversial,  may  be  reviewed  internally,  or 
by  a  combination  of  internal  and  external  peer  review.  Group  discussions 
among  the  reviewers  can  be  very  helpful,  but  individual  reviews  are  easier, 
faster,  and  less  expensive  and  may  be  more  appropriate  at  the  early  stages 
of  a  product’s  development  or  for  products  with  less  impact  and 
complexity. 


Other  Agency  Quality 
Assurance  Reviews 


The  Office  of  Research  and  Development  tracks  how  many  peer  reviews 
are  conducted  each  year  across  epa  and  does  a  qualitative  review  of 
whether  or  not  the  reviews  were  conducted  according  to  epa  policies. 
Managers  and  peer  review  leaders  are  also  expected  to  document  the 
results  of  completed  peer  reviews.  These  results,  in  conjunction  with 
discussions  during  the  peer  review  process,  are  intended  to  help  epa  to 
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ensure  that  the  scientific  and  technical  support  for  its  decisions  withstand 
independent  scrutiny. 


Research  Not  Subject 
to  Review 


epa’s  Science  Policy  Council  Peer  Review  Handbook  recognizes  some 
circumstances  that  might  preclude  the  peer  review  of  a  major  work 
product.  These  include  products  prepared  using  widely  accepted 
methodology;  those  for  which  the  regulatory  activity  has  been  terminated; 
those  with  court-ordered  deadlines  that  may  limit  or  eliminate  time  for  an 
adequate  peer  review;  those  for  which  resources  to  conduct  reviews  are 
limited;  and  products  that  were  previously  reviewed,  if  a  new  application 
does  not  differ  significantly  from  the  original  application. 
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The  following  presents  a  description  of  the  National  Institutes  of  Health’s 
(nih’s)  peer  review  and  other  quality  assurance  review  practices. 

Begun  in  1887,  the  National  Institutes  of  Health  (nih)  has  grown  to  become 
one  of  the  world’s  foremost  biomedical  research  centers  and  the  federal 
government’s  focal  point  for  biomedical  research  in  the  United  States,  nih 
is  one  of  eight  agencies  in  the  U.S.  Department  of  Health  and  Human 
Services’  Public  Health  Service.  Comprised  of  18  Institutes,  6  Centers,  and 
a  Library,  nih  has  75  buildings  on  more  than  300  acres  in  Bethesda, 
Maryland. 

nih’s  mission  is  to  uncover  new  knowledge  that  will  lead  to  better  health 
for  everyone.  To  this  end,  nih  conducts  research  in  its  own  laboratories; 
supports  the  research  of  nonfederal  scientists  in  universities,  medical 
schools,  hospitals,  and  research  institutions  in  the  United  States  and 
abroad;  helps  to  train  research  investigators;  and  fosters  the 
communication  of  biomedical  information.  Research  ranges  from  basic 
understanding  of  biological  processes  and  the  human  genome  to  clinical 
trials  for  ways  to  control  infectious  diseases  and  tests  of  dental  sealants 
for  children’s  teeth,  nih  research  is  used  by  regulatory  agencies  such  as  the 
Food  and  Drug  Administration,  whose  mission  is  to  ensure  that  food, 
drugs,  and  medical  devices  are  safe.  In  fiscal  year  1999,  nih’s  budget  for 
r&d  is  $14.9  billion.  About  82  percent  of  the  budget  will  be  spent  on  grants, 
contracts,  or  similar  awards  to  organizations  outside  the  agency.  These 
awards  comprise  the  extramural,  or  external,  research  program.  The 
remainder  of  the  budget  supports  nih’s  intramural  research  and 
administrative  support  for  both  the  extramural  and  intramural  programs. 

Peer  review  is  conducted  under  authority  of  the  Public  Health  Service  Act, 
as  amended.17  More  specific  policies  and  procedures  are  outlined  in  the 
U.S.  Department  of  Health  and  Human  Services  and  Public  Health  Service 
regulations,  the  Public  Health  Service  Grants  Administration  Manual,  the 
Public  Health  Service  Grants  Policy  Statement,  and  nih’s  manuals  and 
handbooks.18 


Peer  Review 
Definition 


The  Public  Health  Service  Act  states  that  members  of  peer  review  group 
shall  be  individuals  who  by  virtue  of  their  training  or  experience  are 
eminently  qualified  to  give  expert  advice  on  scientific  and  technical  merit. 
The  act  also  includes  provisions  to  ensure  that  there  is  no  conflict  of 


1742  U.S.C.  201,  et  seq. 
1842  C.F.R.  Part  52h. 
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interest  among  the  reviewers.  Agency  officials  said  that  peer  review 
includes  the  expert  evaluation  of  scientific  merit  by  independent 
reviewers. 


p  -d pvj  pw  pra  rpc;  |  Almost  all  research  funded  by  nih  is  peer-  reviewed.  Panels  of  outside 

r  IvtJView  r  IdL.1  C  experts  review  extramural  research  projects  for  scientific  merit  prior  to 

funding  the  research  project.  Institute  directors  have  the  legal  funding 
responsibility  for  both  extramural  and  intramural  research.  Outside 
reviewers  generally  review  the  intramural  research  program 
retrospectively.  Advisory  council  and  institute/center  staffs  also  conduct 
numerous  program-level  reviews. 


Reviews  of  Research 
Proposals 


The  Center  for  Scientific  Review,  formerly  the  Division  of  Research 
Grants,  is  the  focal  point  for  the  conduct  of  peer  review  for  the  external 
research  program.  Referral  officers  review  the  contents  of  some  10,000 
applications  each  grant  cycle  and,  using  written  guidelines,  assign  an 
application  to  a  Scientific  Review  Group.  These  review  groups  judge  a 
proposal’s  or  project’s  scientific  and  technical  merit,  assign  priority 
scores,  and  make  budget  recommendations.  The  specific  criteria  used  to 
assess  the  merit  of  research  project  applications  vary  with  the  types  of 
applications  reviewed.  Criteria  for  grants  include  significance,  approach, 
innovation,  investigator,  and  environment.  Criteria  for  contract  projects 
and  proposals  include  significance,  availability  of  technology  and 
resources,  anticipated  practical  uses  of  the  results,  and  adequacy  of  the 
methodology.  In  addition,  nih  policy  requires  that  all  applications  are 
reviewed  for  the  adequacy  of  their  plans  to  include,  recruit,  and  retain 
both  genders,  minorities,  and  children  as  research  subjects  and  the 
adequacy  of  proposed  protection  for  humans,  animals,  or  the  environment. 

To  ensure  independence,  almost  all  peer  review  is  performed  by  outsiders. 
The  act  and  nih  regulations  require  that  no  more  than  25  percent  of 
reviewers  are  from  within  the  agency;  the  average  is  only  about  1  percent. 
The  Scientific  Review  Groups  are  generally  composed  of  18  to  20 
individuals,  primarily  from  academia,  to  review  as  many  as  60  to  100 
proposals.  The  Scientific  Review  Administrator  recommends  and  the 
Director  of  nih  appoints  review  group  members  from  among  the  active  and 
productive  researchers  in  the  biomedical  community  to  serve  for 
multiyear  terms.  Criteria  for  selecting  the  reviewers  include  demonstrated 
scientific  expertise,  a  doctoral  degree  or  its  equivalent,  mature  judgment, 
balanced  perspective  and  objectivity,  an  ability  to  work  effectively  in  a 
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group,  an  interest  in  serving,  and  an  adequate  representation  of  women 
and  minority  scientists. 

Membership  is  frequently  supplemented  by  temporary  members  and 
written  outside  opinions.  When  a  proposed  research  topic  does  not  match 
a  review  group’s  specialties,  or  when  an  application  sent  to  the 
appropriate  review  group  might  create  a  conflict  of  interest,  nih  may 
convene  a  Special  Emphasis  Panel  to  conduct  the  review.  The  Scientific 
Review  Groups  usually  meet  together  in  person  three  times  a  year  for  2  to 
3  days  but  sometimes  use  teleconferencing.  Mail  reviews  are  used  only  as 
an  adjunct  to  a  full  panel  review,  nih  officials  said  that  by  definition,  peer 
review  recommendations  are  considered  as  advice  only.  However,  while 
other  factors  such  as  maintaining  a  variety  of  research  topics  and  the  need 
to  support  newly  emerging  areas  of  science  are  considered,  most 
extramural  awards  follow  peer  review  recommendations.  Of  about  40,000 
grant  applications  submitted  to  nih  each  year,  up  to  30  percent  are  funded. 

For  each  institute,  a  National  Advisory  Council  mandated  by  the  Congress 
meets  three  to  four  times  a  year  to  conduct  second-level  reviews  of  all 
eligible  grant  applications.19  As  mandated  by  the  Congress,  these  advisory 
groups  typically  include  about  two-thirds  outside  scientists  and  one-third 
lay  members,  such  as  lawyers,  economists,  and  members  of  patient  and 
disease  advocacy  groups.  These  councils  may  also  have  ex  officio 
members  representing  other  federal  agencies.  Councils  make 
recommendations  to  the  Institute  director  about  funding  particular 
meritorious  grants  that  are  seen  as  very  important  but  which  may  not  have 
received  the  best  scores  from  the  scientific  reviewers  and,  in  general, 
ensure  that  the  scientific  peer  review  process  has  been  conducted 
appropriately. 


Reviews  of  In-Progress  The  need  to  continue  funding  projects  over  multiple  years  is  an  important 

Research  criterion  when  deciding  to  fund  new  projects.  In  any  given  year,  only 

about  25  percent  of  the  total  funds  allocated  for  research  proj  ects  is 
available  for  new  projects  that  may  change  the  course  of  a  line  of  research 
or  move  research  into  an  entirely  new  area.  Peer  review  of  ongoing 
external  projects  occur  at  the  time  a  request  is  made  for  renewed  support. 
The  Scientific  Review  Groups  evaluate  these  efforts  with  the  new 
proposals. 


i0Public  Health  Service  Act,  title  IV,  Sec.  406(a)(3)  and  405  (b)(2). 
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Boards  of  scientific  counselors  reviews  the  technical  and  scientific  quality 
of  each  institute’s  ongoing  intramural  research.  Like  the  other  nih  advisory 
committees  discussed  previously  above,  the  boards  are  established  under 
the  Federal  Advisory  Committee  Act  to  review,  discuss,  and  evaluate 
institute,  center,  and  division  research  programs,  projects,  and 
investigators.  The  boards  meet  two  or  three  times  a  year.  The  boards  also 
review  and  evaluate  tenured  nih  investigators  at  least  once  every  4four 
years  and  tenure-track  scientists  must  be  reviewed  mid-point  in  their 
tenure-track  and  prior  to  conversion  to  tenure.  In  addition,  the  boards  may 
choose  to  review  the  work  of  permanent  staff  scientists  or  other 
intramural  scientists.  Chosen  mainly  from  outside  the  government,  board 
membership  includes  internationally  recognized  authorities  in  one  of  the 
fields  of  research  under  review.  For  continuity,  nih  policy  states  that 
members  generally  serve  for  overlapping  5-year  terms,  if  possible. 


Reviews  of  Publication  NIH  administrators  encourage  scientists  they  fund  externally  or  employ 

internally  to  publish  the  results  of  their  work.  Abstracts,  manuscripts,  or 
written  material  by  employees  must  be  reviewed  and  approved  prior  to 
publication.  Typically,  this  process  also  entails  further  peer  review  by  the 
journal,  nih  administrators  can  follow  citation  indexes,  in  which 
peer-reviewed  articles  that  are  cited  are  compiled,  in  order  to  gauge  the 
relevance  and  success  of  funded  researchers. 


Other  Peer  Reviews  To  review  agency  policies,  each  institute  convenes  national  advisory 

councils  with  members  from  the  public  and  from  the  medical  and 
|  scientific  communities  with  expertise  relevant  to  the  nih’s  missions.  These 
councils  may  also  review  and  comment  on  special  initiatives  proposed  by 
the  institute  or,  for  example,  on  research  training  policies.  In  addition,  the 
boards  of  scientific  counselors  provide  evaluation  and  advice  on  scientific 
directions  of  the  laboratory,  tenure  actions  under  consideration,  resource 
|  allocation,  specific  projects  projects,  including  new  areas  of  development, 
and  other  administrative  matters. 


Other  Agency  Quality 
Assurance  Reviews 


In  1996,  the  nih  Director  created  a  Peer  Review  Oversight  Group  to  advise 
on  the  development  and  implementation  of  trans-Nm  policies  to  ensure 
that  the  review  processes  keep  pace  with  current  advances  in  research,  nih 
also  conducts  reviews  of  its  directors,  the  senior  executives  who  manage 
the  agency’s  institutes  and  centers.  In  each  institute  and  center,  Scientific 
Directors  supervise  research.  An  ad  hoc  committee  with  at  least  four 
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members  reviews  each  Scientific  Director’s  performance  every  4  to  6 
years.  This  committee  is  established  by  and  reports  to  the  Institute  or 
Center  Director’s  Advisory  Council  or  Board,  which,  in  turn,  makes 
recommendations  to  the  Institute  or  Center  Director.  Membership  on  the 
ad  hoc  committees  varies  and  can  include  an  Advisory  Council  or  Board 
member,  a  former  Institute  intramural  scientist,  a  senior  scientist,  and  a 
scientific  administrator. 


Rpcparrh  Nnt  QJnhiprt  Occasionally,  administrative  supplements-additional  funds  necessary  to 
rvebedl CILIMOLOUUJ ec l  |  carry  on  approved  work-are  not  peer-  reviewed.  However,  the  project  and 

to  Review  its  administrative  supplement  are  reviewed  if  the  project  is  continued 

beyond  its  original  schedule,  at  which  time  it  competes  with  new  projects. 
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The  following  presents  a  description  of  the  United  States  Geological 
Survey’s  (usgs’s)  peer  review  and  other  quality  assurance  review  practices. 

The  United  States  Geological  Survey  (usgs),  as  the  principal  research 
agency  of  the  Department  of  the  Interior,  provides  biological,  geologic, 
topographic,  and  hydrologic  information  that  contributes  to  the  wise 
management  of  the  Nation’s  natural  resources  and  promotes  the  health, 
safety,  and  wellbeing  of  the  people,  usgs  has  four  research  divisions: 
Biological  Resources,  Water  Resources,  Geologic,  and  National  Mapping 
Program.  The  divisions  support  research  conducted  by  usgs  scientists,  and 
through  competitive  grants  to  external  scientists,  usgs  provides 
information  resulting  from  its  research  that  consists  of  maps,  databases, 
and  descriptions  and  analyses  of  the  water,  energy,  biological,  and  mineral 
resources,  land  surface,  underlying  geologic  structure,  and  dynamic 
processes  of  the  Earth,  usgs’  fiscal  year  1999  r&d  budget  is  $567  million. 
The  four  divisions’  research  spending  for  fiscal  year  1998  was  about 
$545  million  with  about  $507  million  spent  on  intramural  research. 


Peer  Review 
Definition 


The  usgs  defines  peer  review  as  scientific  and  technical  review  by  qualified 
scientific  or  technical  experts  in  the  relevant  discipline(s).  Individuals 
chosen  as  reviewers  are  recognized  experts  in  the  appropriate  field. 
However,  reviewers  are  not  to  be  involved  in  the  preparation, 
development,  or  execution  of  the  program,  project,  or  product  being 
reviewed. 


~r  Rmn  pw  Pra  rt i  ppq  usgs  P°licy  provides  for  peer  review  of  all  research.  In  early  1996,  a  draft 

reel  neview  ridtutes  Department  of  the  Interior  policy  called  for  each  Bureau  to  develop  a  peer 

review  policy.  In  response,  usgs  implemented  a  Bureau-wide  peer  review 
policy.  At  that  time,  the  usgs  divisions  had  review  policies  that  varied  in 
content  and  scope.  The  divisions  were  directed  to  submit  proposed  review 
policies  or  guidelines  for  evaluation  by  usgs’  Program  Council  and 
approval  by  the  Policy  Council.  The  “USGS  Review  Policy”  covers 
(1)  subjects  for  review;  (2)  review  definitions;  (3)  the  management  of 
|  reviews;  (4)  timing,  frequency,  and  documentation  of  reviews;,  and  (5)  the 
departmental  peer  review  guidelines,  usgs  directed  that  the  peer  review 
policy  be  applied  to  all  aspects  of  the  acquisition,  interpretation, 
application,  and  dissemination  of  scientific  and  technical  information. 

usgs’  peer  reviews  are  conducted  to  select  among  competitive  grant 
proposals  and  intramural  projects,  to  evaluate  a  proposal’s  application  to 
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questions  of  public  policy,  and  to  ensure  the  technical  quality  and 
relevance  of  research  at  all  stages  of  development,  usgs’  guidance  to  the 
divisions  allowed  them  to  manage  reviews  in  different  ways  and  at 
different  stages,  depending  on  the  subjects  under  review,  and  required 
them  to  retain  independence  and  rigor  in  the  review  process.  In  general, 
managers  who  supervise  research  programs  are  responsible  for  ensuring 
that  reviews  are  conducted  and  the  findings  are  documented.  The  four 
divisions  have  each  developed  different  peer  review  practices  that  reflect 
usgs’  guidance. 


Reviews  of  Research 
Proposals 


usgs’  policy  states  that  scientific  proposals  for  new  projects  and  programs 
or  for  significant  changes  in  existing  projects  and  programs  will  be  subject 
to  peer  review  or  other  review  as  appropriate  prior  to  final  approval  and 
implementation.  Reviews  must  comply  with  the  Department-level 
guidelines.  The  guidelines  present  factors  that  indicate  the  need  for  peer 
review,  including  large  commitments  of  funds  to  new  or  extended 
research  projects,  communications  regarding  scientific  subjects  of  a 
controversial  nature,  and  major  research  projects  performed  for  usgs 
under  contract.  Determining  the  need  for  peer  review  is  the  responsibility 
of  the  project  or  program  supervisor.  When  peer  reviews  are  conducted,  in 
most  cases,  reviewers  should  be  external  to  the  bureau  and  should  include 
two  or  more  peer  scientists. 

Officials  of  the  Biological  Resources  Division,  Water  Resources  Division, 
and  the  Geologic  Division  said  the  divisions  review  all  research  proposals 
prior  to  funding.  The  National  Mapping  Program  is  planning  to  implement 
a  process  to  review  all  proposals  for  its  research  program  beginning  in 
fiscal  year  1999.  Generally,  the  criteria  applied  by  divisions  to  select 
reviewers  for  project  and  program  level  reviews  are  knowledge  and 
expertise  in  an  area  Divisions’  evaluations  of  the  research  criteria  include 
determining  whether  objectives  are  met,  results  are  relevant  and  timely, 
and  evidence  is  adequate. 


Reviews  of  In-Progress  USGS  guidance  stipulates  that  subsequent  periodic  review  of  projects  and 

Research  programs  are  to  be  conducted.  Ongoing  programs  are  to  undergo  an 

independent  external  peer  review  about  every  5  years,  with  the  number 
and  mix  of  internal  and  external  reviewers  determined  by  the  individual 
responsible  for  the  review.  Each  division  determines  the  specific 
frequency  of  reviews  for  ongoing  projects.  The  divisions’  implementations 
of  the  review  guidelines  have  resulted  in  some  deviations  among  reviews 
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of  projects.  All  four  divisions  review  research  projects  and  programs  while 
in-progress,  but  the  timing  of  the  reviews  differs.  For  example,  all  Geologic 
Division  projects  are  peer-reviewed  annually  after  the  work  has  been 
undertaken,  and  each  program  is  subjected  to  formal  external  review 
every  5  years  by  groups  such  as  the  National  Academy  of  Sciences.  The 
National  Mapping  Program  conducts  quarterly  project  reviews.  The  Water 
Resources  Division  has  a  standing  contract  with  the  National  Research 
Council  for  broad  reviews  of  the  division,  which  address  different  subjects 
each  year.  These  reviews  result  in  recommendations  for  changes  to 
program  direction  and  improvement. 


Reviews  of  Publications  Scientific  products  require  the  Director’s  approval  prior  to  general  release 

and  a  peer  review,  if  appropriate,  is  required.  Qualified,  technical  experts 
in  the  relevant  discipline^)  review  research  products  prior  to 
dissemination.  For  example,  the  Water  Resources  and  the  National 
Mapping  divisions  require  review  of  manuscripts  by  two  peer  colleagues. 
Procedures,  implemented  by  the  Divisions  for  peer  review  of  publications, 
call  for  review  by  scientists  who  do  not  have  any  involvement  in  the 
project.  The  Geologic  and  Water  Resources  Divisions  also  hold  authors  of 
reports  and  other  products  responsible,  through  the  peer  review  process 
associated  with  the  release  of  publications,  to  interact  with  fellow 
scientists  in  addressing  comments,  suggestions,  and  criticisms. 


Other  Agency  Quality 
Assurance  Reviews 


The  “USGS  Review  Policy”  identifies  three  types  of  reviews  that  are  to  be 
conducted  in  addition  to  peer  review:  management  review,  policy  review, 
and  editorial  review  of  publications,  usgs  supervisors  and  managers 
conduct  the  management  reviews  of  programs,  proposals,  projects, 
products,  and  customer  information.  For  example,  these  reviews  are  used 
to  select  and  approve  peer  reviewers  and  types  of  reviews;  ensure 
technical  soundness,  relevance,  and  priority;  confirm  that  objectives  are 
met;  ensure  proper  animal  care  and  use;  and  verify  that  the  media,  style, 
technical  level,  and  content  of  publications  fit  the  intended  audience. 
Policy  reviews  include  review  of  programs,  proposals,  projects,  and 
products  to  ensure  compliance  with  pertinent  policies  and  mandates  prior 
to  receiving  “Director’s  Approval”  for  general  release  of  research 
products.  Editorial  reviews  conducted  by  qualified  personnel  check  for 
expression,  clarity,  mechanical  condition,  organization  and  layout, 
effective  and  appropriate  presentation  and  use  of  illustrations  and  tables 
for  the  intended  audience,  completeness  of  annotation  for  references,  and, 
if  appropriate,  conformity  with  usgs’  style,  usage,  and  format. 
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The  divisions  also  implement  additional  reviews.  For  example,  the 
Biological  Research  Division  has  administrative,  national  program,  and 
science  center  reviews  in  addition  to  the  cited  usgs  reviews.  Most 
Biological  Resources  Division  programs  and  projects  receive  annual  and 
or  “mid-point”  management  and  technical  reviews.  The  Water  Resources 
Division  and  National  Mapping  Program  conduct  numerous  reviews  of 
programs  and  technical  disciplines  that  include  biannual  and  3-year 
reviews.  The  National  Mapping  Program  does  not  require  peer  review  for 
directed  research  projects  being  performed  for  an  external  customer. 
However,  such  research  is  reviewed  for  its  relevance  to  the  goals  and 
objectives  of  the  National  Mapping  Program.  The  customer  reviews  the 
projects,  and  the  contract  or  other  written  agreement  establishes  the 
review  criteria. 


Research  Not  Subject 
to  Review 


All  usgs  research  is  subjected  to  peer  review  or  other  technical  reviews. 
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The  following  presents  a  description  of  the  National  Aeronautics  and 
Space  Administration’s  (nasa’s)  peer  review  and  other  quality  assurance 
review  practices. 

The  National  Aeronautics  and  Space  Administration  (nasa)  conducts 
research  in  earth  science;  space  science;  life  and  microgravity  sciences, 
and  applications;  and  aeronautics  and  space  transportation  technology. 
Within  these  program  areas,  10  NASA  centers  fund  research,  nasa’s  fiscal 
year  1999  r&d  budget  is  $10.1  billion.  In  fiscal  year  1998,  nasa  spent 
$3.5  billion  of  its  $10.5  billion  budget  on  research  funded  through  grants, 
cooperative  agreements,  and  contracts  with  other  agencies,  industry,  and 
academia.  The  remainder  of  the  r&d  budget  funded  procurement  contracts 
for  nonscientific  investigations  such  as  the  development  of  hardware  for 
the  international  space  station,  communication  satellites,  shuttle 
modifications,  and  facilities  construction.  The  funding  mechanism  is 
determined  by  the  research  goal.  For  example,  nasa  officials  said  that  a 
contract  agreement  is  appropriate  for  a  product  or  service  that  will  be 
used  by  nasa.  Grants  and  cooperative  agreements  are  appropriate  when 
research  is  conducted  to  accomplish  a  public  purpose,  nasa’s  research 
may  be  used  to  support  other  agencies’  policy  decisions,  such  as  those 
related  to  regulations  of  the  Environmental  Protection  Agency  or  Federal 
Aviation  Administration. 


Peer  Review 
Definition 


nasa  defines  peer  review  as  scientific  evaluation  by  an  independent 
in-house  specialist,  a  specialist  outside  nasa,  or  both,  of  proposals 
submitted  in  response  to  nasa  research  announcements,  announcements 
of  opportunity,  and  cooperative  agreement  notices.  Peer  Review  is  also 
used  to  evaluate  unsolicited  proposals.  Peer  reviews  evaluate  relevance  to 
nasa’s  objectives,  intrinsic  merit  that  includes  scientific  or  technical  merit 
of  research  methods,  the  researcher’s  capabilities  and  qualifications,  and 
cost. 


p  p  p  •  p  _  pp„  All  nasa  research,  including  research  resulting  from  unsolicited  proposals, 

x  tJtil  rtGVlfc1  W  n  delicti  jg  subject  t0  peer  review,  and  peer  review  is  conducted  primarily  to  award 

funds  on  the  basis  of  scientific  merit,  nasa  officials  said  that  external  peer 
review  is  essential  for  high-quality,  relevant  research,  nasa’s  Federal 
Acquisition  Regulations  Supplements  dictate  that  peer  review  will  be  the 
method  used  to  evaluate  and  select  research  for  funding.20  nasa  is 
developing  a  series  of  instructions  on  the  implementation  of  peer  review. 


2048  C.F.R.,  1835.016  and  1872.403. 
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Currently,  each  nasa  office  works  with  its  own  contractors  that  arrange 
and  manage  the  logistics  of  peer  reviews.  Policies  for  peer  review  are 
documented  through  the  instructions  in  the  research  announcements, 
which  are  written  by  nasa  scientists,  nasa’s  Sponsored  Research  Business 
Activity,  which  is  responsible  for  implementing  research  business  policies 
and  procedures,  has  a  competitive  procurement  effort  underway  to  obtain 
a  single  contract  to  manage  the  logistics  of  peer  reviews  to  gain 
consistency  among  programs  in  the  reviews  of  proposals,  nasa’s  offices 
now  have  five  different  contracts  providing  logistical  support  for  peer 
reviews,  nasa  also  has  efforts  underway  to  develop  uniform  instructions 
for  submitting  research  proposals  and  to  increase  uniformity  in 
development  of  research  announcements. 


Reviews  of  Research 
Proposals 


Intramural  research  conducted  by  nasa  scientists  is  normally  funded 
through  awards  resulting  from  the  same  open  competitive  solicitations 
that  are  used  to  select  extramural  research.  Research  proposals  from  nasa 
centers  and  other  federal  laboratories  are  considered  together  and  treated 
identically  to  those  from  industry  and  academia.  Specific  peer  review 
methods  differ  somewhat  among  nasa  offices  or  disciplines  depending  on, 
for  example,  cost,  resources  brought  to  bear,  or  the  experience  of  the 
reviewers.  Some  offices  have  standing  committees,  some  ad  hoc 
committees,  and  some  a  combination  of  these  for  individual 
announcements.  However,  the  use  of  nasa  experts  to  evaluate  and 
document  findings  of  proposal  reviews  is  universal  across  NASA. 

Peers  include  scientists  from  public  and  private  academic  institutions, 
industry,  government  laboratories,  and  foreign  countries.  Criteria  for  peer 
selection  include  the  research  they  have  conducted,  publications, 
knowledge  and  experience,  and  ability  to  conduct  an  impartial  review. 
nasa  and  peer  review  support  contractors  maintain  databases  of  discipline 
experts  to  identify  peers.  Acknowledged  experts  in  a  discipline  and 
proposal  authors  may  also  suggest  other  qualified  reviewers.  The  officials 
said  that  using  external  peers  ensures  fresh  view  points,  alternative 
perspectives,  and  state-of-the-art  understanding.  The  authors  of  a  proposal 
are  not  involved  in  its  review,  and  peers  are  screened  to  ensure  that  they 
have  no  conflict  of  interest. 

Reviews  are  conducted  by  mail  or  by  panel  meetings,  depending  on  the 
logistics  specific  to  the  review.  Mail  reviews  are  conducted  to  allow  for  the 
selection  of  reviewers  with  very  specialized  expertise  on  technical  and 
scientific  issues  and  technical  approach.  Often  a  panel  review  is 


Page  60 


GAO/RCED-99-99  Federal  Research 


Appendix  IX 

Peer  Review  Practices  at  the  U.s  Geological 
Survey 


conducted  to  reconcile  differences  among  mail  reviews  and  put  the 
proposed  research  in  a  larger  scientific  and  programmatic  context.  In 
fiscal  year  1998,  five  nasa  research  offices  released  a  total  of  57  research 
announcements  and  received  5,048  proposals.  Of  these,  3,778  proposals 
were  peer-reviewed  by  mail,  and  136  panels  met  with  a  total  membership 
of  1,287  peers.  An  official  of  the  Office  of  Space  Science,  which  received 
2,599  proposals,  said  that  his  office  held  91  peer  review  panel  meetings 
throughout  the  year.  Each  panel  consisted  of  7  to  10  members.  NASA 
receives  a  few  unsolicited  proposals,  and  these  are  usually  peer-reviewed 
only  by  mail.  Panel  members  are  solicited  by  telephone  or  letter  and  are 
reimbursed  for  their  travel  costs  to  attend  panel  meetings,  nasa  officials 
said  that  peers  working  on  a  voluntary  basis  contribute  to  the  integrity  of 
the  process. 

Specific  criteria  for  reviews  are  unique  to  each  research  solicitation,  but 
the  criteria  should  be  modeled  after  basic  evaluation  factors,  including 
relevance  to  nasa  objectives,  intrinsic  merit,  and  cost.  Panels  do  not  apply 
overall  scores,  but  rate  each  proposal  on  scientific  and  technical  merit 
using  a  5-point  scale.  In  addition,  nasa  program  goals  and  objectives,  and 
cost-effectiveness  of  proposed  budgets  are  factors  in  the  review  process 
for  individual  proposals.  However,  a  nasa  official  said  that  there  are  wide 
variations  among  nasa  offices  in  the  extent  that  they  rely  on  peer  review 
panels  to  assess  these  factors.  Panels  are  sometimes  asked  to  recommend 
assemblages  of  proposals  that  best  meet  focused  program  objectives. 
Program  managers  weigh  results  of  the  panels’  reviews  against  program 
requirements,  costs,  and  scientific  risk  to  ensure  a  focused  and  well- 
balanced  program.  If  the  nasa  selecting  official  determines  a  proposal  is 
relevant  to  the  agency’s  mission,  it  generally  will  be  funded  if  the  panel 
rated  it  high  in  technical  merit. 


Reviews  Of  In-Progress  Peer  reviews  are  not  normally  used  to  evaluate  in-progress  or  completed 

Research  research,  although  a  yearly  progress  report  is  required  before  a  yearly 

funding  allotment  is  provided  to  the  researcher.  However,  if  a  research 
proj  ect  continues  beyond  3  years,  the  researcher  must  submit  a  new 
proposal  that  is  subject  to  full  external  peer  review  and  competes  with 
new  proposals  for  funding. 


Reviews  of  Publications  According  to  NASA,  the  amount  and  quality  of  the  research  results  are 

judged  by  the  research  community  and  through  publication  in  scientific 
journals  and  nasa’s  Scientific  and  Technical  Information  Program.  This 
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program  manages  and  disseminates  results  of  basic  and  applied  r&d  to 
reduce  unnecessary  duplication  and  improve  the  productivity  and 
cost-effectiveness  of  the  research.  The  program  requires  conformance 
with  review  requirements  prior  to  acceptance  of  research  for  publication. 


Other  Agency  Quality 
Assurance  Reviews 


Project  managers  and  staff  are  frequently  asked  to  comment  on 
management  and  financial  aspects  of  research  proposals,  since  as  a  rule 
science  reviewers  are  not  qualified  experts  in  these  fields.  These 
nontechnical  reviews  are  presented  with  scientific  reviews  to  the  selecting 
official.  There  are  a  few  instances  where  internal  or  program  reviews 
alone  may  be  conducted  for  proposals  that  involve  minor  funding  for 
nonresearch  activities,  such  as  presentations  at  workshops  or  symposia, 
or  a  situation  which  requires  a  quick  response  to  an  unexpected  research 
opportunity.  Normally,  the  program  manager  conducts  mid-point  or 
annual  reviews  of  research  projects  without  input  from  multiple  experts. 
nasa  managers  also  annually  review  required  research  progress  reports  to 
determine  whether  funding  should  be  continued.  Also,  reports  of  findings 
and  new  knowledge  presented  at  the  conclusion  of  a  research  effort  are 
evaluated  internally  to  determine  if  additional  support  should  be  provided 
to  the  grantee. 


A  final  report  that  is  required  at  the  conclusion  of  a  grant  or  research 
effort  must  include  citations  of  all  published  papers  resulting  from  the 
work.  Internal  reviews  of  research  results  are  based  primarily  on  the 
publications  that  have  appeared  in  the  peer-reviewed  literature,  the 
importance  of  the  results  or,  in  some  cases,  the  rigorous  evaluation  of 
results  as  apart  of  international  research  assessments. 


Research  Not  Subject 
to  Review 


All  research  is  reviewed,  through  either  external  peer  review  or  internal 
nasa  review. 
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The  following  presents  a  description  of  the  National  Science  Foundation’s 
(nsf’s)  peer  review  and  other  quality  assurance  review  practices. 

j  The  National  Science  FoundationNSF  is  an  independent  federal  agency, 
with  the  goal  of  promoting  and  advancing  scientific  and  engineering 
progress  in  the  United  States  as  well  as  ensuring  the  nation’s  supply  of 
scientists,  engineers,  and  science  educators.  Of  its  fiscal  year  1999  r&d 
budget  of  $2.7  billion,  $2.5  billion  is  allocated  for  basic  and  applied 
research,  nsf  supports  extramural  research  and  education  in  most  fields  of 
science  and  engineering,  through  about  200  programs.  The  research  is 
funded  through  grants  and  cooperative  agreements  with  almost  2,000 
colleges,  universities,  and  other  research  and  education  organizations 
from  all  parts  of  the  United  States,  nsf  annually  receives  about  30,000 
proposals  requesting  new  or  renewed  support  for  research,  graduate  and 
postdoctoral  fellowships,  and  math,  science,  and  engineering  education 
projects.  About  10,000  new  awards  are  made  annually. 


Peer  Review 
Definition 


nsf  uses  the  terms  peer  review  and  merit  review  interchangeably,  and 
reviews  of  research  proposals  are  “merit  review  with  peer  evaluation.” 
nsf  reviews  involve  knowledgeable  peers  from  the  scientific  and 
engineering  communities  as  the  keystone  of  their  system. 

PEER  REVIEW  PRACTICES 


Research  proposals  under  nsf’s  competitive  grants  program  are 
peer-reviewed  by  experts  from  the  fields  of  research  represented  by  the 
proposal.  Current  peer  review  guidance  is  contained  in  the  nsf  Grant 
Proposal  Guide  99-2,  which  became  effective  October  1, 1998.  This 
document  incorporated  newly  revised  criteria  used  by  reviewers  to 
determine  the  merit  of  proposals.  The  criteria  ask:  (1)  What  is  the 
intellectual  merit  of  the  proposed  activity,  including  importance  of  the 
activity  in  advancing  knowledge  and  understanding;  qualifications  of  the 
author  of  the  proposal;  and  extent  to  which  the  project  suggests  and 
explores  creative  and  original  concepts;  and  (2)  What  are  the  broader 
impacts  of  the  proposed  activity,  including  how  well  the  activity  advances 
discovery  and  understanding  while  promoting  teaching,  training,  and 
learning,  and  broadens  the  participation  of  under  represented  groups;  the 
extent  that  the  infrastructure  for  research  and  education  will  be  enhanced; 
and  the  proposed  benefits  to  society.  Performance  outcomes  from  prior 
nsf  research  grants  are  also  taken  into  consideration.  The  Chairman  of  the 
National  Science  Board  is  charged  with  ensuring  that  nsf  peer  review  is 
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conducted  appropriately,  and  the  Director  of  nsf  reports  annually  to  the 
Board  on  the  merit  review  system. 

Reviews  of  Research 
Proposals 

Proposals  received  for  research  under  the  competitive  grants  program  are 
reviewed  by  a  scientist,  engineer,  or  educator  serving  as  an  nsf  program 
officer,  and  usually  by  3  to  10  other  persons  outside  nsf  who  are  experts  in 
the  particular  field  represented  by  the  proposal.  Authors  of  proposals  are 
invited  to  suggest  names  of  persons  they  believe  are  especially  well 
qualified  to  review  the  proposal  or  persons  they  would  prefer  not  review 
the  proposal.  These  suggestions  serve  as  one  source  in  the  reviewer 
selection  process  at  the  program  officer’s  discretion.  Program  officers  may 
obtain  comments  on  proposals  from  mail  reviews,  assembled  review 
panels,  or  site  visits  before  recommending  final  action  on  proposals. 

Senior  nsf  staff  further  review  the  program  officer’s  recommendations  for 
awards.  The  division  director  receives  a  recommendation  and  decides 
whether  the  proposal  should  be  declined  or  recommended  for  award. 
Normally,  final  programmatic  approval  occurs  at  the  division  level.  Then, 
the  Division  of  Grants  and  Agreements  reviews  the  business,  financial  and 
policy  implications  of  the  proposal,  before  issuing  a  grant  or  other 
agreement.  The  judgments  of  the  peers  as  to  the  extent  that  proposals 
address  the  review  criteria  are  vital  for  informing  nsf  staff  and  influencing 
funding  recommendations,  nsf  relies  on  the  judgment  of  program  officers 
to  make  funding  recommendations  that  address  nsf  strategic  goals. 

Reviews  of  in  Progress 
Research 

nsf  research  grants  are  awarded  for  1  to  5  years,  and  large  ongoing 
projects  (in  terms  of  the  number  of  investigators  involved,  the  time  frames 
of  the  project,  or  the  dollar  amount  of  the  grant)  are  evaluated  by  outside 
experts  who  visit  the  research  sites.  The  reviewers  are  familiar  with  the 
scientists  and  engineers  who  are  conducting  the  research.  The  results  of 
these  reviews  are  used  in  decisions  on  whether  to  continue  funding  of 
these  projects. 

Other  Agency  Quality 
Assurance  Reviews 

nsf  convenes  a  Committee  of  Visitors,  comprised  of  outside  advisors,  who 
review  the  technical  and  managerial  stewardship  of  a  specific  nsf  program 
or  cluster  of  programs.  The  division  director  uses  the  committee’s 
recommendation  as  a  factor  guiding  program  direction.  The  Committee  of 
Visitors  also  assists  the  Foundation  in  its  efforts  to  evaluate  research 
results.  Because  of  the  Results  Act  reporting  requirements,  nsf  is  trying  to 
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track  results  of  the  research  efforts  it  supports,  through  annual  and  final 
reports  on  each  project  and  plans  to  follow-up  for  years  after  the  research 
is  completed  to  identify  impacts  resulting  from  the  research.  Investigators 
receiving  grants  submit  annual  progress  reports  that  are  reviewed  by  the 
program  director. 

At  a  program  director’s  prerogative,  up  to  5  percent  of  the  research  budget 
can  be  used  for  newly  emerging  research  areas  that  are  reviewed  by  nsf 
staff  but  are  not  necessarily  subjected  to  external  peer  review.  This  is 
done  because  the  nature  of  peer  review  to  select  research  for  funding  that 
is  somewhat  conservative  and  may  not  normally  result  in  the  funding  of 
high  risk  or  exploratory  projects,  nsf  believes  it  is  important  to  conduct 
some  exploratory  research  to  further  expand  knowledge  in  certain  areas. 
In  some  instances,  these  grants  are  awarded  for  ideas  that  may  need  quick 
decisions,  such  as  a  case  where  earthquake  information  requires 
immediate  study. 


Research  Not  Subject 
to  Review 


All  research  funded  by  nsf  is  reviewed  through  internal  nsf  review  and,  in 
most  cases,  also  through  external  peer  review. 
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The  following  presents  a  description  of  the  Federal  Aviation 
Administration’s  (faa’s)  peer  review  and  other  quality  assurance  review 
practices. 

Within  the  Department  of  Transportation,  the  Federal  Aviation 
Administration’s  (faa)  mission  is  to  provide  for  the  safe,  secure,  and 
efficient  movement  of  air  traffic  consistent  with  national  security 
concerns.  The  faa’s  Research,  Engineering,  and  Development  Program 
develops  and  validates  the  technologies,  systems,  designs,  and  procedures 
for  the  full  range  of  the  agency’s  operational  and  regulatory  activities.  The 
agency’s  1999  budgets  for  research,  engineering,  and  development  and 
Advanced  Technology  Development  and  Prototyping  is  $202.6  million, 
j  During  the  agency’sFAA  rulemaking  process,  the  Aviation  Rulemaking 
Advisory  Committee,  an  faa  and  industry  group,  discusses  the  confidence 
level  in  the  research  that  supports  a  proposed  rule. 

faa  conducts  research  at  two  research  centers,  the  William  J.  Hughes 
Technical  Center  in  New  Jersey  and  the  Civil  Aeromedical  Institute 
located  at  the  Mike  Monroney  Aeronautical  Center  in  Oklahoma  faa’s 
research  includes  collaboration,  through  a  variety  of  mechanisms,  with 
other  government  agencies,  such  as  nasa  and  DOD,  as  well  as  aviation 
professionals,  industry,  and  universities.  In  April  1998,  faa  reported  that  it 
was  participating  in  292  research  partnerships  with  military,  nonmilitaiy, 
domestic,  and  international  research  organizations.  For  example,  many 
different  research  efforts  are  included  in  faa’s  partnership  agreements 
with  nasa,  the  Department  of  Defense,  and  doe’s  Sandia  National 
Laboratory  to  conduct  research  on  aircraft  safety.  In  some  instances,  faa 
contributes  funding  to  the  partnerships,  and  in  others  it  provides  services 
such  as  facilities  or  staff  expertise.  In  addition  to  these  partnerships,  faa 
funds  research  through  cooperative  research  and  development 
agreements,  which  allow  it  to  share  facilities,  equipment,  services,  and 
personnel  resources  in  cooperation  with  private  industry,  academia,  or 
state  or  local  government  agencies  to  develop  an  idea,  prototype,  or 
product  for  direct  application  to  the  civil  aviation  community.  In  1990,  the 
Congress  authorized  the  Air  Transportation  Centers  of  Excellence,  which 
are  funded  through  cooperative  agreements  with  universities,  to  assist  faa 
in  conducting  research  on  critical  strategic  issues  pertinent  to  developing 
and  maintaining  a  safe  and  efficient  air  transportation  system.  Centers  of 
Excellence  agreements  include  matching  funds  from  nonfederal  sources. 
In  addition,  through  research  grants  and  cooperative  agreements,  the  faa 
supports  advanced  research  in  areas  of  potential  benefit  to  the  long-term 
growth  of  civil  aviation,  in  areas  related  to  the  prevention  of  catastrophic 
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failure,  and  to  counteract  terrorist  acts,  faa  grants  are  generally  awarded 
to  applicants  from  colleges,  universities,  and  other  nonprofit  research 
institutions.  Other  appropriate  research  institutions,  such  as  for-profit 
organizations  and  governmental  entities,  may  also  qualify  to  perform 
research  in  aviation  security. 

Peer  Review 

Definition 

Peer  reviews  are  technical  reviews  performed  by  in-house  and  external 
experts  with  qualifications  equal  with  those  of  the  researcher  whose  work 
is  being  reviewed.  Peers  must  be  knowledgeable  about  the  area  to  be 
reviewed. 

Peer  Review  Practices 

faa  conducts  peer  reviews  of  proposed  research  projects  funded  through 
faa  grants  and  cooperative  agreements.  Through  an  advisory  council,  the 
agency  also  conducts  program-level  peer  reviews  of  planned  research, 
ongoing  research,  and  the  outputs  of  completed  research  for  their 
usefulness  to  faa  and  industry. 

Reviews  of  Research 
Proposals 

The  1990  faa  Research,  Engineering,  and  Development  Authorization  Act 
provided  grant  authority  to  the  agency  for  the  first  time  and  required  a 
review  and  evaluation  process  to  ensure  that  research  proposals  have 
adequate  merit  and  relevancy  to  faa  mission,  faa  officials  also  said  that 
they  are  cognizant  of  the  annual  budget  guidance  from  ostp  and  OMB 
regarding  the  need  to  conduct  peer  review  of  competitively  awarded 
research  proposals,  faa  has  developed  a  formal  grant  application  process. 
Grants  and  cooperative  agreements  are  administered  by  faa’s  Office  of 
Aviation  Research,  and  the  Research  Grants  Program  Office  also  reports 
to  the  Research,  Engineering,  and  Development  Advisory  Committee,  faa 
issues  one  broad  solicitation  each  year  outlining  major  categories  for 
needed  research  and  receives  proposals  to  undertake  specific  r&d  projects 
on  an  on-going  basis.  Each  proposal  is  evaluated  on  its  own  merit,  rather 
than  being  compared  with  other  proposals. 

The  Research  Grants  Program  documentation  states  that  following  an 
administrative  review;  each  proposal  will  be  reviewed  carefully  for  merit 
by  a  technical  evaluation  team  consisting  of  three  or  more  qualified 
people.  A  faa  representative  is  to  be  designated  as  the  team  leader  and  is 
responsible  for  developing  an  overall  rating  based  on  the  ratings  of  the 
team  members,  faa  officials  said  that  peer  reviewers  are  primarily  faa 
employees,  because  they  have  the  required  subject  matter  expertise. 
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Ninety  percent  of  the  proposals  are  reviewed  by  three  individual  peers  not 
involved  in  the  project,  who  are  selected  by  the  program  experts. 
Reviewers  attest  they  have  no  conflict  of  interest.  Reviewers  receive 
guidance  on  the  use  of  four  review  criteria:  intrinsic  value,  relevance  to 
faa  mission,  technical  soundness  of  the  proposal,  and  research 
performance  competence.  However,  a  faa  official  said  that  these  reviews 
are  highly  subjective. 

Reviews  of  In-Progress 
Research 

faa  also  conducts  peer  reviews  of  on-going  research  projects  and 
programs.  For  example,  faa  provides  annual  funding  for  the  Joint 
faa/NASA  University  Program,  a  consortium  comprised  of  the 
Massachusetts  Institute  of  Technology,  Ohio  University,  and  Princeton 
University,  faa  and  nasa’s  technical  experts  conduct  triannual  reviews  of 
the  program’s  ongoing  research  in  areas  such  as  intelligent  flight  control 
systems,  weather  hazard  avoidance,  satellite  navigation,  cockpit  displays, 
and  intelligent  air  traffic  management. 

The  agency  also  conducts  reviews  of  the  work  in  progress  in  its  Centers  of 
Excellence.  For  example,  for  the  Operations  Research  Center  of 
Excellence,  the  agency  and  the  center’s  partners  from  industry  and 
academia  conduct  an  annual  research  review  as  well  as  hold  biannual 
steering  committee  meetings  to  ensure  that  the  research  is  on  track  and 
pertinent  to  the  needs  of  faa. 

Reviews  of  Publications 

The  agency  encourages  intramural  and  extramural  researchers  to  publish 
results  in  journals  that  conduct  peer  review  prior  to  the  acceptance  of  the 
results  for  publication.  The  Civil  Aeromedical  Institute  reported  that  it 
conducts  peer  reviews  of  manuscripts  and  other  work  products  prior  to 
publication. 

Other  Peer  Reviews 

faa’s  annual  Programming  and  Budgeting  Process  results  in  the 
establishment  of  research  priorities  for  six  faa  program  areas  and 
developing  the  r&d  investment  portfolio  to  meet  mission  and  strategic 
goals  and  objectives.  During  this  process,  faa’s  Research,  Engineering,  and 
Development  Advisory  Committee,  which  is  subject  to  the  Federal 
Advisory  Committee  Act,  provides  peer  review  of  the  planned  research  at 
a  program  level,  faa  established  the  committee  as  a  result  of  the  Aviation 
Safety  Research  Act  of  1988,  to  provide  advice  on  research  needs, 
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objectives,  plans,  approaches,  content,  and  accomplishments.21  The 
Federal  Aviation  Administration  Research,  Engineering,  and  Development 
Authorization  Act  of  1990  expanded  the  committee’s  responsibilities  and 
set  the  membership  to  not  more  than  30  representatives  from  research 
Centers  of  Air  Transportation  Excellence,  universities,  corporations,  user 
groups,  associations,  consumers,  and  other  government  agencies.22 

During  the  annual  process,  subcommittees  of  the  advisory  committee 
review  faa’s  proposed  research  and  research  that  is  already  in-progress 
and  make  recommendations  about  priorities  and  merit  of  the  research. 
Evaluation  criteria  for  assessing  research  project  descriptions  include, 
among  others,  Results  Act  considerations,  congressional  mandates, 
mission  relevance,  research  outcomes  and  outputs,  and  whether  the 
project  plan  describes  a  credible,  well  understood,  work  effort. 
Documented  procedures  are  used  to  develop  merit  scores  assigned  to 
proposed  research  project  descriptions. 


During  the  annual  programming  and  budgeting  process  faa  target  area 
teams,  which  include  internal  research  sponsors  from  the  six  program 
areas,  are  heavily  involved  in  prioritizing  and  planning  all  research  efforts 
and  in  decisions  about  what  research  to  fund.  In  addition,  faa  program 
staff  conduct  reviews  of  some  research  programs  once  or  twice  a  year,  or 
as  needed.  For  example,  a  faa  official  said  that  program  review  teams 
include  a  faa  Associate  Administrator,  Department  of  Transportation 
representatives,  and  faa  stakeholders.  On  an  ad  hoc  basis,  faa  solicits 
assistance  from  external  organizations  such  as  the  National  Academy  of 
Science  to  review  its  research  efforts. 

Officials  at  faa’s  two  laboratories  described  their  own  review  processes,  in 
addition  to  the  annual  programming  and  budgeting  reviews:  The  Technical 
Center  conducts  mostly  applied  research  and  supports  extramural 
research  through  contracts,  cooperative  agreements,  and  partnerships. 
Most  of  the  research  directly  supports  faa’s  regulations.  The  Center  has  no 
documented  peer  review  procedures,  but  the  Results  Act  has  made  the 
Center  more  conscious  of  needing  a  process  to  look  at  quality.  The 
|  cCenter  has  technical  reviews  of  in-progress  research  by  two  groups.  A 
group  of  mostly  retired  government  and  industry  experts  in  aircraft  safety 
conduct  about  six  reviews  a  year  of  the  quality  of  the  research.  In  addition, 
ad  hoc  groups  of  industry  experts  advise  faa  about  duplication  of  research 


Other  Agency  Quality 
Assurance  Reviews 


21P.L  100-591,  Nov.  3,  1988. 
22P.L.  101-508,  (Nov.  5, 1990). 
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between  faa  and  industry  or  opportunities  to  share  data  from  related 
research.  The  Civil  Aeromedical  Institute  conducts  intramural  research, 
and  reviews  are  primarily  internal.  The  Institute  employs  some  of  the 
world’s  best  scientists  in  its  narrow  fields  of  research,  so  the  number  of 
outside  experts  is  limited.  Internal  reviews  are  conducted  of  Aeromedical 
Research  Resumes,  which  describe  technical  details  of  proposed  research 
projects.  In  addition,  an  Internal  Medical  Research  Group  of  faa 
laboratory  and  headquarters  staff,  and  one  outside  expert,  meets  four 
times  a  year  to  review  research  proposals  and  quarterly  progress  reports. 

Research  performed  under  procurement  contracts  is  reviewed  during  the 
annual  programming  and  budgeting  process.  After  this  process  establishes 
research  needs  and  budgets,  faa  technical  staff  prepare  statements  of 
work  for  research  they  want  to  perform  under  contracts.  These  same  staff 
review  subsequent  proposals,  and  after  contracts  are  awarded,  they  are 
expected  to  monitor  the  researcher’s  performance.  In  addition,  contract 
deliverables  are  reviewed  by  internal  experts  prior  to  acceptance  by  the 
agency. 


Research  Not  Subject 
to  Review 


All  faa  research  is  reviewed  by  faa,  its  industry  stakeholders,  or  others. 
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